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outcomes, and validity assumptions. The research methodology is 
described with attention to the sample, the validation model, 
longitudinal and cross-sectional approaches, the input of students 
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attention is focused on outcomes of the college experience and the 
relationship of outcomes to the world of work. Major study findings 
are surnmar i zed , and dissemination of the results is discussed. 
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PREFACE 



During our dissemination at conferences and professional meetings of some preliminary findings 
based on results described in detail in this report, many colleagues expressed interest in the 
broader issue of validation. Concerned about validating their own programs, they wished to 
know how we conceptualized the validation of a liberal arts, outcome-centered curriculum. 
We responded to this interest by describing faculty questions that stimulated the initio >*udy 
of college outcomes at Alverno and by summarizing research results based on ongoing vacuity 
questions. We explained that these questions and results were placed within the context of a 
validation model that has guided our efforts at Alverno College for the past seven years. 

This report follows that same approach. First, this overview and summary describes the 
rationale for validating outcome-centered higher education curricula and our validation model 
drawn from faculty questions. We then describe the research objectives based on tnese questions 
and our approaches to instrumentation and methodology. Then we synthesize major 
conclusions from ten research studies that follow this overview and summary. We relate the 
findings to our overall purposes and discuss implications for higher education. We also include 
abstracts of the research r orts, identify our dissemination strategies and list the range of 
colleges and universities, porations and schools whose questions and insights have con- 
tributed to our efforts during these past seven years. 

This report represents the collaborative work of the Alverno faculty, Office of Research and 
Evaluation staff, Alverno students and alumnae, and Milwaukee organizations and professionals. 
Our acknowledgments to them and to our other colleagues follow the preface. 

This work is dedicated to our students, whose belief in our ability to improve education gives 
us the faith and courage to continue learning, and to research the penetrating questions that 
challenge higher education today. 

Marcia Mantkowski 
Austin Doherty 

Milwaukee, Wisconsin 
May 1983 
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ABSTRACT 

What differences Joes college make? Can it really promote 
the kind of broad personal and intellectual development that 
lasts a lifetime? Can it enhance a person's abilities and 
improve his or her chances at having an effective career? Can _o 
benefit the "new" student body adults, women, minorities --- as 
well as traditional college students 9 Do the outcomes of college 
show up on the job? 



That students change in college is taken for granted by nost 
college educators and has been demonstrated by several 
researchers of college outcomes (Astin, 1977; Feldman & Newcomb, 
1970; Heath, 1977; Pace, 1979; Vaillant, 1977; Winter, McClelland 
?t Stewart, 1981 ). That students change in college as the result 
of performance in a particular curriculum is more difficult +o 
show. How students change, and who changes and why—and v 4 th 
respect to what broad, complex abilities, learning styles and 
cognitive-developmental patterns (Chickering & Associates. 1981; 

is even more illusive. Demonstrating that these changes 
persist beyond college to effective performance in work and 
personal roles is perhaps most challenging of all. Shewing that 
abilities selected by college faculty and demonstrated by their 
students are used by outstanding professionals in the world of 
work, is clearly a new issue for college educators. 



] The faculty we have been working with are our colleagues at 
Alverno College, a midwestern liberal arts college for women 
with about 1400 degree students in both weekday and weekend time 
frames. Alverno, which ht+? focused for a century on preparing 
women for professional careers, formally adopted an outcome 
centered approach to its curriculum in 1973, accrediting 3tudeiits 
for progressive demonstration of certain broad abilities in all 
subjec!" areas. 
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Yet these are precisely the issues raised by one liberal arts 
college faculty who broke with tradition and implemented an 
outcome-centered liberal arts curriculum in W5. The faculty 
identified broad outcomes promised by many colleges. But they 
defined these complex abilities through a set of pedagogical 
levels that allowed for their successive achievement, and created 
measures that assessed not only knowledge, but the student's 
performance. When the first students were about to graduate from 
the new curriculum, the faculty en^a^ed in a multi-faceted 
attempt to focus on the external validity of the abilities they 
had identified. Tt was in the context of an overall plan to 
validate outcomes of college that Alverno designed a set of 
parallel and interrelated research studies. These studies were 
then funded by a major three-year grant from the National 
Institute of Education toward the ^oal of establishing the 
validity of abilities learned in college for later careering and 
professional performance (Mentkowski ft Doherty, 1977). The 
••esearch studies, proposed in p ive project objectives, dealt with 
two themes. One is related ta identifying broad outcomes of 
college, including those abilities critical to effective 
performance at work, and how abilities cap, be defined, assessed 
and validated. Another theme concerns the extent to which 
college contributes to development and change in outcomes, 
particularly if they are defined as cognitive-developmental 
patterns, learning styles and broad, generic abilities. 

We had a distinct advantage in desi^ninfl and carrying out 
research on these, issues. The faculty, with whom we were 
working, had already identified the more "intangible" outcomes of 
college such as life span development and lifelong, independent 
learning, as important Roala. They had spent several years 
Lientifyin^ the broad, generic abilities they wanted their 
graduates to show (e./>. f communications, analysis, social 
interaction, problem solving and valuing; Alverno College 
Faculty, 1976 and relating then in increasingly explicit terms 
to the program, courses and learning activities their students 
en^a^ed fn. These " abilities were defined as developing (or 
t^achabl^)', as transf err i r\t> across multiple settings and an 
Internalized cha rac t*»r ist i -'S of th»- p^.rr^n, rather than discrot.p 
sets of sk i 1 1 .=5 . 

This p;ave us a full ran^e of no 1 i ♦ vro-.^nnora t ed definitions to 
work with. The college's own methods for assessing each 
student's progressive development of her abilities (Alverno 
College Faculty, 107 f 0 provided one set of measures for those 
outcomes. In* our first project, objective, we contributed to 
identifying and validating a set of cross-disciplinary measures 
■>?' Jplle^e performance ( Assessment ':omm i t tec/Of f ice of Research 
•,nd Evaluation, VW), I'lfP, Friedman, Mnnt.ko wsk i , Doutsch, 

'hovar S Allen, V)M, M ; Friedman, Mm f;ko wsk i , Karley, I.oaoker * 
')\ 1UM0). 
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Along with these definitions and measures, we identified a 
parallel"' set. These were drawn in part from other practitioners 
and researchers with whom we were already working. While there 
were few, if any, measures that matched the faculty defined 
abilities directly, we selected measures representing the newer 
directions for defining and assessing broad, more intangible 
college outcomes (Chickering, 1981; McClelland, Winter H Stewart, 
1981), because these were most like the overall goals of the 
Alverno curriculum. Measures were selected that most nearly 
reflected the faculty's . emerging' theory of performance 
assessment. For the second project objective, we administered a 
battery of twelve cognitive-developmental, learning style, and 
generic ability measures — we call them human potential measures 

to over 750 students in a five year cross-sectional and 
longitudinal study. Our goals were to describe change in 
college, to see if change could be attributed to performance in 
the curriculum, and to identify the underlying themes in these 
change patterns (Mentkowski * Strait, 1983). We also thereby 
contribute to the development and further test of these measures 
(Mentkowski, Moeser & Strait, 198'5). 

At the same time, we set about systematically gathering data 
about the students' perspective for the third project objective. 
This meant creating an open-ended interview format that allowed 
students to generate their own definitions of the college 
experience, with particular emphasis on how they saw themselves 
changing, and why. We administered the interview to the same 
group of 80 stude-ts at the end of each year in college and to 
about 40 Seniors. These students were already part of the larger- 
sample just described, and were completing the human Potential 
measures in that longitudinal study (Much S Mentkowski, 1982). 

To examine outcomes in the workplace and other post-college 
life settings, we used several approaches. For the fourth 
project objective, we first extended our interview studies beyond 
graduation. Over 30 two-ye^r alumnae, also interviewed as 
Seniors, completed indepth interviews where they discussed new 
learning at work, and the abilities and processes that enabled 
eareorinp and professional performance after college. Second, we 
seated a careering questionnaire for all 60 two-year alumnae 
We were able to focus specifically on how new graduate and 
rwo-year alumna attitudes and expectations evolve as they develop 
their professional roles and make career decisions since the same 
measure was concurrently administered to Seniors (Mentkowski, 
Much A f.ienoke-Holl, 1183). For the fifth project objective we 
initiated two studies with 1R0 outstanding professionals in 
nursing and management, to derive models of the actual abilities 
they perform on the job, in order to compare these with the 
outcomes sought by the college as well as those described by its 
rraduat-s (Ment.kow.iki, Do Back, Bishop, Allen A Hlanton, 19HO, 
Mentkowski, O'Brien, McRuehern H l-'owler, 
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We are rinding some encouraging results: 



• the verifiable outcomes of a liberal education 
in college include broad, complex processes of 
trie kind educators have traditionally claimed; 

• they include cognitive-developmental patterns, 
learning styles, intellectual abilities and 
the more active/interactive abilities sought 
in professional work situations, and qbiliti.es 
related to the broader domain of personal 
devel opment ; 

• student development of these cognitive-developmental 
patterns, learning styles and abilities can be 

m en s u re d and v al i d a t e d ; 

• student development of these abilities can be 
related to successful performance in a specific 
curr ionium; 

• both traditional students and "new students" of 
v a r v i n g a g es and life b a c k t \ r o u n d s show p o 1 1 e r n e d 
development, of th»*se obiliti.^s; 

• g radua tes co n t i nue to d evelon these ab i 1 i t ies and 
adapt them into their later work and life settings 
a f ter col 1 ego; 

• these abilities can be related directly to those 
used on-the-job by effective professionals. 



*rfe have also been abl^, along the way, to make some 
contributions! to the newly developing field of educational 
program evaluation and to the repertoire of procedures for 
validating developmental outcomes. In many ways, we had to 
rethink the operational definition of validity ns it is applied 
in a prnetioed-basod research sotting. In addition, our work 
seems to be offering some substantive support for the goals of 
outcome-centered curriculum 1 



These ore early results from :m effort that is now ongoing 
and a port of the learning process. Hut they do suggest that 
higher education von indeed hr 1 p society achieve its equal access 
and mobility goals by contributing demonstrably to each student's 
cognitive , in ten personal , and personal / professional growth 
abilities. They indicate that :nilege olso contributes lo the 
student's ability to integrate these abilities and anply them 
effectively in later life settings, particularly in the worll of 
wo rk . 
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WHY FOCUS ON OUTCOMES IN HIGHER EDUCATION? 

Why specify outcomes in higher education? Society as a whole 
is currency questioning if outcomes claimed by higher education, 
especially the liberal arts, are actually achieved. This . is 
parUv because societal needs for higher education have changed. 
Se nave become a' service rather than a production society where 
lore and more of us need a quality education and technological 
skills We have become a knowledge society, where the emphasis 
is on using knowledge because we can no longer master it all. 
Ind we have become a society of rapid change, where each person 
needS preparation for changing jobs and responsibilities. 
Employers of college graduates complain that equates no longer 
hav'e traditional outcomes of college such as thinking wr ting 
and problem solving, let alone the ability to adapt skills to 
changing roles and contexts. 

Periods of economic stress sharpen the demand for usefulness. 
There is more emphasis on showing that abilities learned in 
college make a difference in contributing to society after, 
"liege. Consequently, higher education is expected to show a 
reaUonship between abilities learned in college and 
professional pr ductivity and development. Education for work 
has become a new theme on college campuses. 

In the past, college as preparation for life was generally 
ass-med Hiphly selective colleges admitted persons with high 
scores on admissions tests and were rarely asked to demonstrate 
that their graduates had productive lives after college because 
ie colege outcomes showed that income, status, and 

1 Hv tv the work force were more pronounced among college 
Tra a s he new student body has changed that. Minority 
eSnic and racial groups, the poor, the handicapped, and women 
now making up a larger segment of the college popula ion. 
ihev are coming to college expecting higher status jobs, 
veonomic and social mobility are thought to result from higher 
e cat on. Minorities expect that college will assist - casing 
discrimination and allow them greater access to soc^ys 
tenet its. Hontrad it ional students, adults who are 

need in multiple roles, are also coming to college in 
record numbers. They expect that college learning oes indeed 

unencumbered by the need to earn a living or to support a fam,.l.y. 

Expecting that abilities learned in college will "reotly 
,onf bute to one's opportunities and success at work comes 
, r cuTarly from the large non trad 1 1 ional college population. 

p particularly women, have already experienced the 
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impact of lack of 'abilities that are needed for advancement out 
of traditional clerical and service positions to positions with 
more responsibility and opportunity. These women are likely to 
expect, ask for and demand that college be accountable to 
demonstrating that the educational outcomes taught are those 
necessary to achieve their own professional goals (Cross, 19^1). 
And t rad i tional age s tuden ts a re now joined in this e xpec ta t ion 
of a career after college (As tin, 1902 )• 

Higher education faculty question higher education's ability 
to respond to these needs. They ask if liberal arts outcomes can 
survive in the new aura of learning for work rather than learning 
for its own sake. Can liberal arts goals be developed in an 
atmosphere of professional education and education for work? 
Will open access lower standards? Will the more traditional 
outcomes of college be sacrificed for graduates' technical 
expertise? Are students still learning to analyze, to think 
critically, to solve problems, create new ideas and ways of 
thinking, to appreciate multiplicity in context and culture, aud 
to achieve quality of life? Can colleges be responsive to the 
now student body and the values of today 1 s student? 

Faculty also question how liberal arts colleges can maintain 
an orientation to the demands of society to teach toward 
careering and the needs of the marketplace and still maintain the 
"student centered" atmosphere of the liberal arts college. Here 
student development is a primary outcome and focus. College Is a 
time to find one's way out of ndolescence and to take on adult 
responsibilities, or to broaden one's world view through the arts 
n nd humani ties. 

College students in general are also pressing for the more 
intangible outcomes of college. Self- fulfillment has been 
labeled as the "new moral ity" in our society ( Yankelovieh, 19B1 ) 
and college students are also expecting their efforts to bring 
self- fulfillment and personal development. Sol f- fulfillment is 
clearly a goal. College students also expect advancement an! 
career achievement (As tin, l^rt^). 

All of society seems to be more interested in accountability. 
The consumer movement, the rise of special interest groups, are 
two indications that individuals are expecting institutions to be 
more accountable, to complete their share of the contract. 
Colleges have been known to promise economic and social mobility, 
personal growth, and other broad outcomes. Students are more and 
more taking them at their word, and expecting that the degree 
makes a difference. They are asking colleges to demonstrate, not 
just to promise, that the institution will be responsible for 
fulfilling their part of the bargain. Kxpeo ta t ions for quality 
of life, for careering after college, for preparation for life as 
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well as work, for achieving personal development as well as 
professional development, all create an atmosphere of 
accountability. 

These concerns for quality of education under conditions of 
equal access, equality of educational opportunity, 
accountability, a demonstrated relationship between college and 
personal and professional development, and personal growth are 
expressed by students, higher education faculty and by society at 
large. They have prompted the move toward defining, assessing, 
and validating outcomes. 
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WHY' VALIDATE OUTCOMES IN HIGHER EDUCATION? 



We have ju$t outlined the rationale for focusing on outcomes 
in higher education- Why establish the validity of outcomes? 
Most colleges and un i versi ties have no t felt compell ed to 
validate that students achieve outcomes or to relate those 
outcomes to future outcomes. Indeed, the effectiveness of 
college has often been taken for granted. Why launch a major 
effort to validate outcomes? 

The rationale for validating outcomes is similar to that for 
focusing on outcomes. First, the press for accountability in 
higher education is logically translated in to demonstrating that 
education is related to and is adequate preparation for work, and 
that education is adequate preparation for life. Validating the 
outcomes of college means demonstrating that a liberal arts 
education assists students to meet the prerequisites for later 
personal and professional performance. But the press for 
accountability is not just a utilitarian one. We are no longer 
interested in demonstrating only that education is useful. We 
are interested in demonstrating that education is equitable, that 
persons without traditional backgrounds can achieve traditional 
outcomes. Demonstrating that our open access policy does not 
lower quality is important, as is demonstrating our response to 
accoun tabil ity . 

Still more important for the adult student is the need to 
demonstrate that outcomes achieved can be attributed to the 
college experience rather than just to maturation. Does college 
enhance life experience for the older adult, or does education 
interfere, rather than build on experience? We are no longer 
willing to accept that outcomes demonstrated at graduation are 
valid unless they persist over time, or contribute in some way to 
the development of later abilities that are critical to future 
outcomes (Astin, 1977, p. 210). 

If outcomes are no longer defined as static, but as 
developmental, then change and its causes are important aspects 
of demonstrating *val idity. What curricular aspects cause change 
in higher education? This question shows an increased emphasis 
on the importance cf continued program development. It is 
generally recognized that embarking on validation research can 
enhance higher education* s ability to create ef fee tive 
programming. Focusing on abilities and processes as outcomes, 
rather than knowledge alone, and assessing for them in a 
performance-based curriculum, is a "new idea" in higher 
education. Consequently, it is expected to prove itself — to show 
that it is doing what it claims to do. New strategies are 
usually much more open to question and expected to be researched 
before adopt ion. 




Validation research can do much to enlighten us ,on the 
characteristics of the now student's learning, abilities, 
cognitive-developmental patterns and learning .styles so that all 
of higher education, whether or not it is performance-based, oan 
become more responsive to student needs* How do students learn? 
: low do they develop? And how do students actually experience 
learning—from their point of view? Validation research is 
critical to building a general izable (educational model for adult 
learning and development, particularly in view of the needs of 
today's more non traditional student body. 

Finally, validation research that identifies the abilities of 
effective professionals bridges the gap betwee.n the college 
faculty and tne professional community. Both groups have a stake 
i a insuring that ab il it ies learned in pro fessional prog rams are 
those critical for effective performance at work after college. 
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DEFINING, ASSESSING AND VALIDATING COLLEGE OUTCOMES 



In response to recent concern about the value of a liberal 
arts degree, college educators are beginning to identify, measure 
and credential broad abilities that are expected outcomes of 
colleg^ (Laacker 8 Palola, 1981). Moreover, some . college 
educators are no longer satisfied to judge program effectiveness 
by comparing their students' performance against standardised 
test norms. Rather, they are questioning how colleges might 
assess students ; using criteria or standards derived from outcomes 
describing the'* liberally educated , competent adult . Other 
educators view college as a catalyst for lifelong development, 
and want to know if abilities learned in college are related to 
the future personal and professional performance of graduates 
(Mentkowski & Doherty, 1977). 

These educators are interested in comparing students' mastery 
of broad abilities to their potential for enhanced human 
development. How do outcomes characteristic of college students 
compare with their developmental potential, with what is possible 
for them to achieve as humans? Some educators feel these 
questions should be raised not only about learned abilities 
faculty can currently measure and credential, but also about the 
more "intangible" outcomes of the college experience, those 
traditionally promised to graduates by most liberal arts 
colleges. These more intangible outcomes include continued life 
3pan development, transition to "life after college," transfer of 
learning to various settings and professional positions, 
self-directed and integrated personal functioning and lifelong 
learning. *\ 



Educators are beginning to define and asgess for broad 
generic abilities or competences, and more intangible outcomes. 
Their goal is to further define and understand the nature of 
abilities and outcomes they teach toward as an important source 
for curriculum development. One problem these educators face is 
the lack of standardized external criterion measures that measure 
Abilities and that predict later performance after college, to 
which they can compare student performance outcomes. There has 
been more interest in operational understanding of broad outcomes 
since publication of The Am erican College (Sanford, 1962), and 
the recent move toward outcome-centered curricula is a thrust in 
that direction (Grant 4 Associates, 1979). 

In the recent past, some educators, colleges and professional 
schools have identified outcomes and developed ways to assess 
them (Grant & Associates, 1979; Lo acker, 1981). (Examples 
include Alverno College, Antioch School of Law, Brigham Young, 
College III of the University of Massachusetts, College for Human 
.Services, Delaware County Community 'College, Florida State, 
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Harvard University, Iowa Wesleyan, Kirkhof College, Mary College, 
Metropolitan State, Mt. Hood School of Nursing, New Roche.lle 
College," North Adams State, Northwestern University School of 
Music, Our Lady of the Lake, Southern Illinois University School 
of Medicine, University College at the University of Louisville, 
University of Montana School of Law, the University' of New Mexico 
School of Medicine, and others.) Many of these institutions are 
now addressing outcome validation issues. They are asking hard 
questions about the extent to which students are able to 
demonstrate outcomes educators have identified as important for 
all college students to manter. "But what measures are available 
that will contribute to program evaluation and outcome 
validation? 



Several efforts in defining and assessing college outcomes 
are specifically focused on performance measures of general 
abilities and characteristics predictive of effectiveness in 
later life (e.g., ACT's College Outcome Measures p roject, McBer 
and Company's Cognitive Competence Assessment Battery). These 
no re focused measures might appear redundant with the usual grade 
reports and standardized achievement or uptitude tests in 
predicting future performance. Yet these conventional measures 
and indices have not shown much relationship to later behavior 
(McClelland, 197"?, 1980). The effectiveness of the new 
performance measures has not been determined as yet, but initial 
test3 are underway in this study and elsewhere (Winter, 
McClelland Stewart, 1981). 

In 1975 , the Fund for the Improvemen t o f Pos t-Secondary 
Education supported a consortium of colleges in trying out some 
newer measures to assess outcomes. As a member of this group of 
colleges, Alverno participated in the FIPSR project, awarded to 
M c Be r and C om pa ny , by a d m i n i s t e r i ng so m e o f the s e n e w . mea su re s . 
These instruments, collected or developed by McBer, later became 
known as the Cognitive Competence Assessment Battery (Winter, 
McClelland A Stewart, 1981). 

When Alverno sought to identify external, criterion measures 
for inclusion in a validation study of student outcomes, we 
selected these measures because they most nearly represented some 
of the abilities, identified by A.lverno faculty. The Cognitive 
Competence Assessment Battery provided a particular focus on 
generic abilities of analysis, and included assessment of motive 
dispositions and other cha rue t.e r i s t ics important to the 
rel n t ionshi p between learning and later behavior. Because other 
colleges wore also administering those measures, we could count 
on some compar ison d ata . 



Li nk i ng Fd uc a t i on and Work : 
Generic Ability Mea s 1 1 re s 




These newly-developed measures of generic abilities can serve 
as better outcome measures, but we are still faced with the need 
to measure abilities learned in college in the context of 
lifelong learning and development. How are abilities learned in 
college transformed through personal and professional experience? 
How can we recognize them in the older adult? The search is on 
for better ways to measure the more intangible outcomes of 
college, those that are often referred to as personal development 
outcomes (Bray, Campbell & Grant, 1974) or other personal 
maturity variables (Heath, 1974, 1976, 1977). How else might we 
insure that college outcomes become integrated aspects of the 
whole person that might be expected to develop beyond college? 

Developmental Theory: 
Cogni tive-Developmental Measures 

It is in relation to the problem of defining and assessing 
abilities learned in college set within a context of lifelong 
learning and life span development that we proposed using 
cognitive-developmental theorists' descriptions of human growth 
and development as sources for college outcome measures 
(Mentkowski * Doherty, 1977). Indeed, Chickering and Associates 
(1981 ) have recently advanced the argument that an overarching 
goal of higher education is encouraging developmental change. 
Developmental psychologists have described broad developmental 
domains that can be measured, such as moral ' development 
(Kohlberg, 1976); ego development (Loevinger, 1976); cognitive 
developme . (Piaget, 1972); and intellectual and ethical 
development (Perry, 1970, 1981). These theorists provide us with 
descriptions of. the way in which individuals cognitively 
structure meaning and make sense out of their experiences. 
Descriptions of development, whether via a series of stage3 
(Piaget, Kohlberg), ego levels (Loevinger), or positions (Perry) 
provide us with a partial picture of students' potential for 
growth. They describe some of the more universal outcomes of 
human functioning against which educators can validate more 
intangible curriculum outcomes. 

While we do not expect that educators will use a student's 
current developmental level, position or stage as a measure of 
performance to credential or pass a student, such information can 
he used to describe where the student is in his or her 
development. Assessing student performance on these measures 
over time gives us important information on individual patterns 
of development during college, and helps us evaluate the extent 
to which college or specific curriculum interventions are 
contributing to, the general cognitive growth of learners. 

This approach to validating student outcomes suggests 
assessing students on various levels of cognitive development as 
part of program evalua tion designs . Using 
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cognitive-developmental rneasc^es to assess college outcomes has 
another Important value.. The results can6 be useu to inform 
instruction, and to assist in creating appropriate curricula. We 
expect this research to reduce the "gize of the existing gap 
between developmental theory and educational practice" (An tin, 

nas). 

Experiential. Learning Theory: 
Learning Style Measures 

Experiential learning theory and research has more recently 
described learning as a process (rather than as static outcomes), 
where knowledge is created and linked to action through the 
transformation of experience (Kolb, in press). While 
cognitive-developmental theories describe assimilation and 
accomodation as the basis for an " interactive learning process, 
these theories are less likely to describe individual differences 
in learning. Cognitive-developmental patterns tend to describe 
common paths in the growth of intellectual development. A 
variety of reseachers have centered on learning style as an 
important indicator of student learning and development ( Curry, 
1983). - Basically, these approaches are interested in specifying 
individual differences in approaches to learnings cognitive 
styles, and differences in learning style preferences. Since 
feedback *on learning style is one way to assist students to 
analyze their own approaches to' learning, faculty find that 
learning style measures can be important not only for curriculum 
design," but also for .assisting students to become more open to 
other modes of learning ('Deutsch & Guinn, Note 1). The Council 
for the Advancement of Experiential Learning has supported 
development of teaching and assessment strategies "based on 
learning by experience (Keeton ft Tate, 1978), and giving credit 
for-learning that occurs in other than formal, or classroom 
learning settings. Experiential learning is seen as a process- 
that links education, work and personal development (Kolb, in 
press). We have proposed using learning style measures as a flay 
to tap college outcomes particularly because Alverno's curriculum 
Ls based partly on experiential learning theory (Doherty, 
Mentkowski A Conrad, 197B), and because of the strong emphasis on 
student involvement in both in- class and off-campus learning 
experiences. 

Competence Assessment: 
Performance Interviews and Inventories 

Another approach to the definition and assessment of outcomes 
we researched in the current studies was the performance 
assessment of effective professionals in order to build models of 
their abilities or competences. While performance assessment of 
alumnae is rare, we determined it to be a way to identify 
abilities alumnae do perform after college, to establish a /link 
ho abilities learned during college. Performance assessment of 
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alumnae was beyond the scope "of the current set of studies until 
we had first completed the roufid of open-ended perspectives 
interviews and careering questionnaires (see below). We do plan 
future alumnae studies using performance interviews. For the 
current study, we did use performance interviews to assess the 
competences of outstanding professionals. In addition, we v 
employed performance characteristics inventories which enable a 
3tudy of professional perceptions of the abilities, competences 
and .behaviors descriptive' of outstanding versus average 
performers. We selected the approach of Job Competence" 
Assessment developed by McBer and Company (Klemp, 1979; 
McClelland, 1976) to build professional competence models, 
because the underlying definition of abilities or competences an'd 
principles of assessment most nearly matched that of the Alverno 
facul ty. 



Perspectives on Learning and Careering : 
Interviews and Careering Questionnaires 

The outcomes of college also need to be described from the 
student's perspective. Clearly, development of college outcomes 
measures focused on abilities acquired during college and 
expected to be related to performance after college, that 
describe intellectual and personal growth across the lifespan, 
and performance assessment of professionals cn-the-job, is just 
getting underway. Measures of cognitive-developmental patterns 
have been used primarily for research purposes, and measures of 
learning styles, while many and varied, have little experience as 
college outcomes measures. 



It seemed imperative, then, to take a path initiated by Perry 
( 1970) in the sixties, that of conducting open-ended interviews 
to discover how students experience college. We proposed, 
conducting broad, in-depth longitudinal interviews with students 
to tap their perceptions, and to thereby gain some insight into 
the determinants of the outcomes of college from the student's 
point of view. We also expected to uncover some of the 
individual differences in learning patterns and the several paths 
that students take during college to achieve their goals. We 
hope to expand our understanding of who benefits from college and 
why, and what kinds of experiences characterize students in a 
perfqrmance-based or outcome-centered liberal arts curriculum. 
Further, the interviews could serve as a context for interpreting 
result? from the human potential measures, and for seeking the 
links between abilities learned in college to those demonstrated 
after college. While some of these research goals go beyond 
those reported here, thir approach is effective in raising 
further research hypotheses and for cofnmunicat ing the nature of 
student change to faculty. 
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We developed' careering c questionnaires to assess student, 
alumna and professionals 1 careering and professional development. 
Th/se questionnaires allow collection of demographic data, 
information on pail and unpaid employment, careering history, and 
attitudinal information. '\Caree~r Lag questionnaires also collect 
data on a range of variables that provide a context for the 
performance and perception studies of professionals. 

Ma t c hi ng F rain e works a nd j^easjire. s _fr o _Curr i- c_ ula r J^oal s_ 
~a~nd Cessment ^rino^pljss 

A primary reason for undertaking evaluation and validation 
studies of student outcomes in college is to inform continued 
curriculum development. This includes' more clearly specifying 
outcomes, lea'rning strategies, assessment techniques and 
evaluation methods. Educators are working to develop curricula 
that repond to the students 1 learning styles, that capitalize on 
the adult's range of experiences . and that reflect y .what is 
understood so far about patterns of younger and older adult 
development and learning. But this effort will succeed only if 
we question the selection and effectiveness of current frameworks 
and corresponding college outcomes measures for college 
curricular settings. , * 



Clearly, selection of frameworks, and corresponding 
instruments as external criteria or standards against which a 
college examines its ability to facilitate student growth is 
appropriate if there is: O a match between the goals and 
objectives, of the college and the framework used, 1 and 2). a match 
between the college's principles of assessment and the theory of 
assessment used to, develop instrumentation based on the 
framework: 

Instruments which have been used for theory testing— even 
though they have demonstrated reliability and validity—need 
to be filtered' first through the practitioner's goals, 
objectives, learning strategies and assessment processes. 
Once they emerge from . this crucial dialectic, they may be 
effective' program evaluation instruments as well (Mentkowskl, 
1QB0, p. 28). 

Therefore our practice-based research using any of the 
measures to establish the validity of college outcomes nreds to 
be understood in the context of their use. This context at 
Alverno College Includes • a philosophy of education, an 
outcome-centered curriculum and principles of assessment which 
have been in the process of development by Alverno faculty for 
over ten years (Alverno College Faculty, 197'), 1977, 1979). 
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DEFINING AND ASSESSING OUTCOMES AT ALVERNO COLLEGE 



How Do We Define Outcomes? 



Alverno' s faculty are concerned with defining and assessing 
outcomes of college. The student's continual development is at 
the center of institutional goals. Thus, the major outcome of 
college is growth or change. Faculty expect college to be a 
significant and positive facilitator for student growth, and a 
catalyst for lifelong learning and development. Rather than 
thinking of college as a cause and student growth a3 an effect, 
growth is a result of the interaction between a self-directing 
individual who plays a role in initiating her own growth, and a 
learning process. Both faculty and student select and involve 
her in learning which challenges and supports personal change. 
The role as learner continues after college throughout" the life 
span, and learning becomes a means by which she realizes her 
potential for professional development and personal growth. 

This emphasis on growth of the person across the life span, 
for which college is a catalyst, determines what broad outcomes 
are identified. Yet any definitions of outcomes need to retain 
the breadth 'and complexity characterized by college-level 
learning and performance. The college takes responsibility for 
contributing to growth and development of lifelong learners, and 
for learning in college that continues after college. Such goals 
are broad, and a commitment to them provides a philosophical base 
far a faculty working collaboratively to develop a curriculum. 
But ultimately, a faculty- needs to define these broad, more 
"intangible"' outcomes of college if they are to teach and assess 
for them. 



What are the developmental, holistic and generic abilities 
each student must demonstrate in order that faculty consider her 
a lifelong learner? At Alverno, the focus on 1 outcomes took shape 
in 1971 when the faculty, in a yearlong series of intense faculty 
institutes, struggled to respond to the questions, "What should a 
student get out of spending four years with us?", "What kind of 
person did we hope she would become?" and "How are we'helping it 
to happen?" As the year progressed, it became clear that a focus 
on outcomes a liberal education challenges the individual to 
develop, needed to be accompanied with questions about the 
definition of abilities, the nature of. the learning experiences 
provided, and the way in which abilities — we called them 
competences — could be assessed (Alverno College Faculty, 1976). 
For the next two years, an academic task force synthesized the 
many abilities the faculty identified into eight general outcomes 
and defired each as an ability or competence. Each was then 



What Are the Abilities or Competences? 
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further define:] via a senuer. tial , ine r^as ingly complex set of six 
levels. The competences ; i:v: 

• Oomm m i eat ions 

• Analys I. s 

• Problem Solving 
9 Valuing 

• Social In te rae t ion 

• Taki ng Respons Lbil i ty for the Rnvi ronmen t 

• Involvement in the Contemporary World 
e Aesthetic Response 

All students are expected to progressively demonstrate levels 1 
to 4 of each ability, usually by trie end of the general education 
sequence. She then demonstrates levels r ) and 6 of those 
abilities most related to her major and minor areas of 
concentration. 

Faculty have defined the meaning of each ability or 
competence, the sequence and increasing complexity of the 
competence levels, the relationship of each competence level to 
other levels and to other competences as well as the 
relationships across academic disciplines in the Facu lty Han dbook 
on Learning and Asse ssment (Alverno College Faculty, 19777. 
Instructional methods are suggested. Each competence level also 
describes the criteria for assessment, and suggests appropriate 
instrument stimuli and modes (with an emphasis on production 
tasks) for assessing performance. At Alverno, college outcomes 
are defined as abilities or competences considered to be complex 
'processes. Faculty define abilities as dev elopmental, hoi is tic 
and generic (Alverno College Faculty, 1979). 

Developmental Abil ities 

For an ability or competence to be d eve l o pmen i_tal means that 
it is teachable. Thus, the ability or competence is broken open 
into sequential descriptions or pedagogical levels that describe 
increasingly complex elements and/or processes which are acquired 
by : students over time as the result of instruction and where each 
Level requires a more complex demonstration of the ability. 
Further, competences that are developmental continue to change 
after college, as additional learning experiences contribute 
toward developing greater complexity. 

Holistic Abi J, i ties 

For. an ability to be hoi is tic means that each developing 
ability involves* the whole person. Complex abilities or 
competences include a behavioral component, a knowledge 
component, an affective or self-perception component, as well as 
a motivation or disposition component (Klemp, 1979). All or most 
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of the components of a competence or ability can be inferred from 
an observable demonstration or performance. Traditionally, 
colleges have required demonstration of only the knowledge 
component. When competences or abilities are defined 
holist Lcally , then knowledge, skill, attitudes, sel f- percept ion 
and dispositional components are specified. Within a particular 
context, abilities or competences can then be defined as 
observable behaviors. These components are expected to become 
integrated, and together with other abilities, involve the whole 
person . 



Generic Abil ities 

For an ability or competence to be gen eric means that the 
developing, holistic ability will transfer across situations and 
settings. Thus, abilities are defined as transferable. The 
kinds of situations to which abilities are expected to transfer 
include those a student encounters in exercising multiple roles. 
Generic abilities are expected to transfer not only to situations 
in college and work, but also to personal arn professional 
situations after college. Generic abilities equip students with 
skills that transfer from one situation to another, across roles 
and positions within a particular occupation, and even across 
o ecu pa t ions. 



Most students will ultimately be taking on different roles 
simultaneously. The abilities they acquire in college are 
expected to assist them not only in their professional roles, but 
in personal roles such as citizen, family member and parent after 
college. Professional roles, as well as the personal ones, 
continue to change and develop. Acquiring abilities that are 
developmental, holistic and generic assumes that students become 
learners in college and become self-directed in learning how to 
learn. Learning how to learn consists of learning strategies 
that make up the concept of "lifelong learner." We expect that 
our studies of student and alumna perspectives on learning and 
careering will help us define these broad concepts. 



How Can We Develop These Abili ties ? 

How can faculty develop these abilities in each student so 
abilities become internalized, integrated and generalizable? In 
1973, the faculty implemented an "outcome-centered" curriculum 
and developed learning methods to teach toward the competences 
(Alverno College Faculty, 1977). The curriculum emphasized 
assisting the student to develop these abilities in ways that are 
unique to her own individual differences in learning 'style and 
how she conceptualizes learning. Learning strategies build on 
the theory of "experiential learning" (Doherty, Mentkowski # 
Conrad, 1978). The experiential dimensions of the curriculum 
have been expanded such that students in each of the 18 academic 
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and professional depar tments are immersed in opportunities to 
experience the constraints of the work world by engaging in 
nentored off-campus experiential learning (OCEL) where 
transferring anilities learned in college Is paramount. 
Classroom learning experiences likewise focus on student 
Involvement in learning situations where concrete experiences, 
reflection, conceptualizing ideas and concepts, and plans for 
motion are tested out in new performance situations. 



How Win ye Know a Student 
H a s Ac h 1 e v ed _Th ese A hi n tj_es ? 

low will faculty know a student has achieved these abilities 
or competences according to their prescribed criteria or 
standards? The 'issessmen't process developed by the Alverno 
faculty has been described elsewhere (Alverno College PacuMy, 
1979), and represents one of the more recent directions in 
reconoeptualizing assessment ( V/il lingham , 1980). The assessment 
process is patterned in part on assessment center technology 
(Moses S Byham, 1977). Alverno relies on volunteer assessors 
from the Milwaukee business and professional community to judge 
effective student performance, as well as the faculty who design 
instruments and judge performance both in the Assessment Center 
and through courses. 



Four fundamental principles of assessment are specifying 
criteria, relying on multiple judgments, choosing from alternate 
^r7o~rm~ance modes so as to elicit the full range of the 
developing " ability, using expert judgm ent to infer student 
abilities from this performance, and providing the student an 
opportunity for sel f-assessment. 



Criteria 

Once outcomes are defined as abilities or competences', 
assessing for them means defining the criteria for judging 
student performance. Thus, faculty have defined abilities or 
competences not only by the competence levels, but also by 
specifying assessment criteria. 



An important characteristic of assessment is that of 
evaluation of student performance in relation to criteria or 
st andards ( criterion- referenced) , in contrast to students 
perfo rming relative to norms ( norm- referenced) created just from 
the range of performance of other students. While standards are 
informed by the range of student performance, they are also open 
to input from other sources (e.g., descriptions of abilities or 
cognitive patterns from theories of learning and development; 
abilities that characterize effective professional performance). 
Identifying appropiate criteria or standards is a difficult task 
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and worth a research agenda 
results reported in this 
with this task. 



of its own (Glaser, 1981). The 
paper are in tend ed to assist faculty 



An issue that arises when specifying criteria is the 
relationship of the criteria to the abilities one is measuring, 
and also the relationship of those abilities to broad and 
inclusive college outcomes such as "lifelong learning," "reaching 
one's full potential," "becoming an independent learner," 
"developing critical thinking" and "learning to learn." 
Abilities students must perform in order to graduate, as defined 
through assessment criteria, can be distinguished from broad 
outcomes that are more intangible. Educators may agree on these 
more intangible outcomes and may consciously use them as 
frameworks in teaching. They may even assess for them 
diagrrosfcically in many ways. And faculty have used these 
outcomes to select external criterion measures to validate the 
outcomes of college. But educators do not demand evidence from 
student performance assessments in order to graduate students, 
nor do faculty guarantee such outcomes. 

Specifying criteria for assessment is a faculty effort to 
make the more intangible outcomes of college, and defined 
abilities or competences, operational. Faculty work to identify 
both specific and broad criteria for judging student performance 
at a particular competence level. For each broad ability to be 
assessed, faculty must make the ability explicit through criteria 
so students can understand what performance is required. 
Therefore, faculty need to describe the ability sufficiently 
through criteria statements such that it can be reliably and 
validly assessed. At the same time, the complexity of the 
abilities assessed limits how explicitly these criteria are 
stated. Criteria for assessing student performance of abilities 
fall on a continuum from broad to specific. Thu^. , assessment 
calls for multiple, expert judgment by faculty. 



M'O t| pie Judgments 

Alverno faculty also recognize that any one sample of student 
performance is just that — a sample of what the student is able to 
do in a given context, in response to a particular instrument 
stimulus. Consequently, Alverno faculty rely on multiple 
judgments. This means observing her performance cumulatively, in 
a number of contexts, acros3 a number of settings, across time, 
and across a variety of performance modes. 
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Alternate Performance Modes 



An important challenge in defining criteria for assessment is 
to require that students demonstrate not only the knowledge 
component of abilities, but also demonstrate the behavioral, 
dispositional and self-perception components. Learning to do as 
well as to know puts the emphasis on learning how to perform, and 
requires that the performance mode match, as nearly as possible, 
the ability being assessed. 



Because of the complexity of the competences being assessed, 
faculty design instruments complete with stimulus and performance 
mode 'and criteria) that elicit to the fullest extent, the 
student 1 s developi ng ability. Thus , Alverno faculty have 
committed themselves to designing assessment techniques that 
employ production tasks rather than reco gnition tasks. That is, 
the student ilTTequired to generate a response to an instrument's 
stimulus, rather than simply to indicate recognition of 
information. Consequently, faculty are likely to employ 
performance modes such as essay, group discussion, oral 
presentation, interview, and in-basket , rather than modes such as 
multiple choice, short answer, true-false, etc. Performance 
modes* are designed requiring the student to demonstrate behavior 
similar to the ability as usually expressed rather than an 
artificial mode (e.g., to demonstrate Social Interaction skills, 
she would perform in an actual group discussion). 



Expert Judgment 

Use of production tasks requires expert judgment, defined as 
special knowledge or skill ("expertise") that the assessor brings 
to the judging situation and applies in a rigorous or disciplined 
way. In the context of higher education, where faculty teach 
toward sophisticated abilities, complex cognitive structures, and 
highly skilled performances, faculty are accustomed to the use of 
expert judgment in instruction and assessment. Expert judgment, 
which involves the use of inference in abstract analytical 
thinking,' is basic to assessing student performance at advanced 
levels. Expert judgment is a practical instructional and 
assessment tool and is in constant use by faculty in higher 
education who insist on production tasks to assess performance. 
A treatment of issues surrounding the- use of expert judgment can 
be found in Mentkowski, Moeser and Strait (1983). 



Self-Assessment 

Self-as3essment, or student assessment of her own 
performance, her perc eptions of the extent to which her 
performance meets criteria,"' is ;in important component of the 
assessment process. Assessment provides a challenge that assists 
the student to take responsibility for her own learning, to 
assess herself, and to become more self-directed. 
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Assessment of student performance leads to evaluation and 
revision of instruments and clarification and further development 
of criteria for assessment. Faculty work to continually clarify 
and develop criteria so as to specify both specific and generic 
criteria for credentialing student performance. 

These charac teristics of assessmen t are important to 
recognize because they have implications for the selection of 
Q xternal criterion measures for validating the faculty defined 
outcomes of college, and for realizing our project objective to 
validate Alverno assessment techniques. 

jjhat Are Student Outcomes of the Learning Process? 



Since out comes are very generally defined as growth or 
change, and are visible as change in performance, ability or 
competence definitions communicate what the student does or 
performs, rather • ' than wha t the faculty does or performs. Note 
that college outcomes include self-assessment, or change in the 
student's perception of herself as a learner and as a growing, 
changing individual. Ia addition to student performance, student 
perceptions are equally valuable outcomes of college. 

What Are Alumn ae Future Out comes? 

Because faculty define college outcomes in relation to the 
student as lifelong learner, faculty also seek to define future 
outcomes, to attempt to "see" and conceptualize outcomes that 
develop from those demonstrated in college. Future outcomes help 
provide a picture of abilities as they appear "full grown." They 
orient faculty toward defining outcomes of college in ways that 
describe the beginning of abilities as they are taught in 
college, in relation to those graduates will need five, ten or 
even twenty years after college. Abilities needed for the future 
are built on abilities taught in college. Analytical thinking 
expressed by deriving a hypothesis from a set of interrelated 
studies for a biology class may be quite different from the 
inductive, problem finding analysis an environmental specialist 
use3 on the job. College must educate students for the future, 
not just for the present. Analytical thinking defined for 
college learning must be related to post-college roles to insure 
future personal and professional outcomes. Yet we know very 
little about what those relationships are. 

Future outcomes also include student expectations for 
realizing career and professional opportunities, expectations 
that an investment in college will contribute to adequate 
preparation for performing in professional situations, 
realization of self- fulfillment, and an enhanced quality of life. 
Beyond student expec tationa, faculty expectations for students 
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include and expanded rol^ as a lfiarner who can mak* the 
t. rans i t i on p min rol le/^- ho w«>r\ :in1 t.<> 1 i [V a Tier col 1 pfv*. 
w «iriiltY want Indent.? to l>* »i : j >in ■: • s»* i f-d i ree t"d learners and to 
work toward achieving persona! and pro f<>ssional r ^oal integration 
(Barley, Mentkowski. c < ftohaf'r, 1980). 



What A r « t f i e C orn pon_ f l n 3 °S 1 
Lea rni nt> Process ? 

The six questions underscored above are repeated in Figure 1, 
a graphic of faculty questions and learning process components. 
They set the statfe for the development, in 1976, for the faculty 
focus on est abl ishi the validity of the outcomes of college. 



LEARNING PROCESS 



What are the developmental, holistic 
and generic abilities each student 
must demonstrate in order that we 
consider her a lifelong learner? 



COMPETENCES 



How can we develop these abilities 
in each student so they become 
internalized, integrated and 
generalizable? 



EXPERIENTIAL 
LEARNING 



How will we know if each student 
has achieved these abilities according 
to our prescribed standards? 



ASSESSMENT 

PROCESS, 
TECHNIQUES 



What are the outcomes of the 
learning process, those credentialed 
and those expected but not 
credentialed? 



STUDENT 
CHANGES IN OUTCOMES 



What are a student's realizations 
in perception and professional 
performance as a lifelong iearner? 



ALUMNAE 
FUTURE OUTCOMES 



Figure 1. A Description of Alverno Learning Process Components. 
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ESTABLISHING THE VALIDITY OF OUTCOMES 



Validation studies can he "an important source for insight 
about how human beings learn and develop. Educators are urgently 
seeking the best available frameworks for understanding what and 
how their varied students learn, which experiences stimulate and 
fanhance that learning, and how that learning fits into the tasks 
of lifelong growth. Such studies are also designed for 
verification demanded by the need for accountability. Basically, 
validation helps to focus four kinds of questions which are asked 
by educators as well as by the consti tutenc ies they serve: 

• Descriptive questions: "What is occurring?" 
"How is it occuring?" 

• Ascriptive questions: "Why is it occurring?" 

• Evaluative questions: "Is what is occurring 
'good' compared to a criterion or standard?" 
"Is the standard val id?" 

• Prescriptive questions: "What should 
be occurring?" 

As the new science of program evaluation ha3 emerged, it has 
become apparent that existing resources for establishing validity 
(e.g., American Psychological Association, 1974) are not 
sufficient to the task. of validating developmental outcomes. Nor 
is the controlled-experiment model on which these approaches are 
predicated either appropriate or possible in a dynamic, 
interrelated practice setting (Bryk, 1933; Cronbach & Associates, 
1980; Parlett 4 Hamilton, 1976). 

Like several o ther investigators (Grant, Note 2; Messick, 
1980; Popham, 1978), we have therefore opted for a validation 
approach geared to the unusual complexity of the learning 
outcomes involved in college, as well as to the fluidity of 
program and population that characterize college instruction. 
Several features represent our attempt to respond more 
effectively to the constraints and opportunities of validating 
developmental outcomes in a dynamic program. 

■ In education, a main criterion for demonstrating validity is 
showing that changes in student performance over time occur as 
the result of college. In contrast, the validity of the end 
product alone rather than how it developed can be important in 
noneducat.ional settings. In the work world, employers <nay only 
be interested in selection or retention of employees or in the 
extent to which a candidate for promotion can demonstrate an 
ability, not how or whether the ability was acquired at the 
organization or whether the ability can or should be taught. 
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Trie way in which a person acqui res an ability is critical for 
educational programs. How persons learn, and how tTiey develop 
outcomes is important information for enhancing the quality and 
effectiveness of programs. Wha t causer, change? If college can 
b »? 3a id to f a c L 1 i t a t c ha ng ^ i a s t ud e n t pe r f o rmance , then t h e 
1 ea r n L a g p r 0 r e s : * c a n h e s a i d to b o v a 1 id. 

Es tabl ishing PValua tion/Validation 
an a ^om ponen t of the Learning Process 

In 1976, Mverno faculty made a commitment to establish the 
validity of outcomes. They ilentified several major questions as 
their initial thrust, and .1 assigned" an eight -year plan for 
carrying out the research objectives operational ized from these 
questions ( MeatVio wsk i , 1Q77 b) . 

To carry out these research questions, the facul-y first 
created a context for validation by establishing evaluation as a 
concept and function, and created an Office of Research and 
Evaluation. Evaluation/ validation is thus a part of the 
learning process (Figure 2). Establishing evaluation/validation 
as a curricular component led to the identification of the 
following five research questions. They are: 

0 Are the competences and assessment 

techniques of the learning process valid? 

• flow do students change on college outcomes 
described by their potential for cognitive 
development, 1 earning styl es , and generic 
abil ities? 

• Are outcomes mirrored in students' 
perceptions of their learning and abilities? 

• How do outcomes learned in college relate 
to 1 if elong learning, abil it ies , careering 
and professional development after college? 

• What competences describe the performance and 
perceptions of outstanding professionals? 

Each of these questions was 0 perationalised via an overall 
validation design, complete with specific questions, designs, 
instruments, and methods so a more systematic validation of 
outcomes could occur. During the past seven years, from 
1976-1983, these qyestions have been researched with support from 
Alverno College £nd from a three year grant from the National 
Institute of Education. 
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LEARNING PROCESS 



What are the developmental, 
holistic and generic abilities each 
student must demonstrate in order 
that we consider her a lifelong 
learner? 



How can we develop these abilities 
in each student so they become 
internalized, integrated and 
generalizable? 



How will we know if each student 
has achieved these abilities according 
to our prescribed standards? 



Is the learning process we use to 
develop and assess for abilities 
actually working the way we have 
designed it? 

Are changes in performance of 
student outcomes related to college 
instruction? What is the relationship 
between current outcomes and 
future outcomes? 

How do current and future student 
outcomes compare against internal 
and external standards? 



What are the outcomes of tb r 
learning process, those crede>itialed 
and those expected but not 
credentialed? 



COMPEl 


fENCES 






EXPERIENTIAL 
LEARNING 






ASSESSMENT 

PROCESS, 
TECHNIQUES 

-- 







EVALUATION/VALIDATION PROCESS 



STUDENT 
CHANGES IN OUTCOMES 



What are the students' realizations 
in perception and professional 
performance as a lifelong learner? 



> 



ALUMNAE 
FUTURE OUTCOMES 



Figure 2. A description of Alverno program components with 
evaluation/validation process . 
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A main outcome of the research is the overall approach to 
validating outcomes-, that emerged from researching the five 
objectives. It is appropriate here to describe this approach, 
the features of our attempt to validate outcomes, and the overall 
validation design to provide the context for the ten research 
reports that follow this overview and summary. 

Identifying Assump tio n s About Val id i ty 

During our ongoing dissemination of the issues and earl;; 
results described in this report, many of our colleagues in 
higher education were interested in a broad overview of how we 
conceptualized validating a liberal arts, outcome-centered 
curriculum as a first step in thinking about the validity of 
their own programs. In order to define "validity" as a concept 
and create a framework for establishing validity of abilities 
learned in college, and to communicate this to our colleagues, we 
set forth our assumptions about validity that were identified as 
we researched the five project objectives stated above. 



Ya 1 i at ion Is Developmen tal 

When we create programs, we assume that the program will 
continue to develop. We recognize that most educational programs 
are undergoing various changes, and that new programs have 
start-up time and may then undergo periods of maintenance. "But 
if a program is dynamic and responsive to students, further 
change will continuously occur. 

Where a program is in its development is critical to the 
type3 of strategies used to demonstrate its validity. The kinds 
of internal and external criteria or standards to which a program 
is held depends on the extent to which faculty have defined 
outcomes and are able to assess them, the availability of 
information from which standards can be drawn, and also on how 
long the program has been in operation. It is hardly conceivable 
to fault a program for not having related student outcomes to 
future outcomes if the program is new and does not yet have 
alumnae with extensive post-college experience. If faculty 
define competences or abilities (rather than grade point average 
or subject area tests) as outcomes, and few theoretical 
frameworks for understanding these competences exist, one cannot 
fault them for not establ ishing 'construct validity. If there are 
no tested measures of college outcomes available, one cannot 
fault them for selecting new and untried measures as external 
criterion measures. 



Thus, the kinds of validation questions and issues that can 
be addressed by a faculty concerned with validating outcomes is 
limited to a degree by how far the faculty has come in 
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conceptualizing and implementing tr« curriculum, and by what 
measures are available for comparison. This is an especially 
important consideration in validating performance-based liberal 
arts curricula .since they are generally of recent vintage. 
Indeed, our own attempt to begin validation research coincided 
with the anticipated graduation of our first students from our 
performance-based curriculum, three years after its 
im pi em en tat ion. 



Validation is an Illuminative, Diagnostic Process 

Establishing the validity of college outcomes is never 
"finished." Since programs change ^nd continue to develop one 
cannot and should not consider a program ever • completely 
validated, Further, validation strategies are applied to a 
complex system. Each aspect or level in the system is 
interrelated with another aspect and level, and every change 
changes everything. As validators, we face a considerable 
challenge in trying -to weigh the effectiveness of such integrated 
.environments and their elements. Couple this with an increased 
emphasis on standards rather than normative comparisons, and it 
is clear we face an enormous complexity in validating outcomes. 
How we approach this complexity — our "mindset" — will impact our 
ability to influence the future evolution of higher ■ education 
(Mentkowski, 1980). Because of the complexity of context of rcqst 
programs in higher education and the complexity of abilities and 
outcomes toward which one is teaching, validation efforts cannot 
"prove" validity, but can illuminate the quality and 
effectiveness of programs and the extent to which changes in 
student outcomes are related to future outcomes. 



To justify the amount of time, effort and resources required 
for validation research, results pust be diagnostic. Validation 
results must be usable to improve programs and to continually add 
to the insight faculty bring to teaching, learning and assessment 
issues. Establishing validity means to continue, throughout the 
life of the program, to engage in efforts to bring one closer and 
closer to realizing program goals and objectives, which also 
change . 

Validation Relates Theory to Practice 
and Research to Evaluation 

When Lawrence Kohlberg initiated the Just Community approach 
to schooling, he made the leap from theory to practice. This 
step allowed a test of concepts emerging from his theory and 
research studies, and contributed to their credibility for the 
educational world. Some years after this leap to practice, 
Kohlberg confessed to the "psychologist's fallacy" (in Hersh, 
Paolitto, * Riemer, 1979) of assuming that developmental theory 
as exemplified by stages of development could or should form the 
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most important cornerstone of educational practice. This 
theorist's fallacy has Lty counterpart in the "researcher's 
fallacy, " in which we are tempted to assume that the goals, 
methodology *n;l instrumentation that are characteristic of 
r n-earch studies seeking theory development ar.d demonstrating 
-ause- effect relationships should form the cornerstone of an 
aoproach to the practice of evaluation and validation. 

While many program evaluation studies in current literature 
seem to depend almost entirely on the techniques of the 
researcher, evaluation has begun to emerge as a separate 
discipline. Evalua tors have evolved strategies that clearly 
recognize differences between the purpose of research studies and 
those of evaluation, and have created alternate approaches (Bryk, 
1983; Parlett S Hamilton, 1976). This development, as well as 
the growing recognition that practitioners are equal partners in 
creating theory and practice (Mosher, 1977), sets the stage for 
avoiding the "researcher's fallacy. 11 



A mindset for program evaluation thus begins with the 
awareness that evaluation goals 0 and strategies are better 
selected and deriv?d from th,e practitioner than from the, 
theorist. The question is not "What is aviilable that we can use 
to validate 9 " Rather, "How might we best analyze the special 
characteristics of this curriculum so that our validation 
objectives match the nature of .the specific program? What is the 
relationship between tools for assessing broad outcomes of 
.college and instruments that assess the defined abilities from ^a 
program?" In the previous section we have described Alverno's 
curricular goals and theory of assessment so that a rationale for 
selecting the frameworks and instruments we^ used to validate 
outcomes* could be critiqued. One projected result of this move 
from theory-to-practice and. from research-to-evaluation is that 
we seek to investigate questions suggested by practitioners, and 
to consider the context in which validation is attempted. 



Validation is Contextual 



Earlier, we commented on the importance of recognizing 
validation as a developmental process that walks "hand in hand 
with the program its methods are applied to. Clearly, the 
context in which validation research is conducted has several 
important implications for validation designs and strategies./ 

First, we conceptualize validation in an ongoing, changing 
curriculum where the- object of study does not "hold still." 
Second, validation goals and objectives need to be derived from 
curricular goals and objectives which ultimately are refined 
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through the validation process. The philosophy underlying the 
curriculum, beliefs about how students learn, and student and 
faculty roles impact the kinds of validation objectives and 
strategies that can be employed. This need not be taken as a 
negative constraint. Rather, if we are to avoid the researcher's 
fallacy, then "validity" of validation strategies means that we 
design validation goals and strategies within the context of a 
particular setting. The press of the setting can often serve as 
a guidepost and beacon in validating nontraditional outcomes. We 
benefit from such an approach later when results from validity 
studies are ready to be discussed, criti mod, and ultimately 
im pi erne n ted . 



Third, the design for validating outcomes needs to flow from 
the structure characteristic of the context. Validating outcomes 
cannot be successfully initiated if the way in which outcomes are 
defined is not considered. Involving faculty and students in 
validation strategies cannot occur unless expectations set for 
their involvement are apparent in the program itself. For 
example, students who "come to understand the need for multiple 
assessment of their abilities are more likely to understand why 
they are asked to perform on other than faculty designed measures 
(Mentkowski, 1979). Again, rather than being perceived as a 
constraint, the context should be seen as the source for design 
and implemen tation guidelines. The "validity" cneck of the 
context is an important indicator of the extent to which the 
results from validation studies are those that are both tri'e and 
useful . 



Establishing the validity of programs is a relatively new 
concept. Sets of standards (Rossi, 1982) for conducting program 
evaluations have been formulated, and these standards contain 
norne operational advice. The evaluation research community has 
edited a number of volumes to aid colleagues. But this thrust 
has been a recent development. 



One source of definitions of validity is set forth by the 
measurement community for instrument validation. These types of 
validity have become one way in which the field of educational 
measurement can identify measurement techniques that will yield 
valid, reliable scores from which valid inferences can be drawn. 
These standards define validity as establishing content validity, 
face validity, construct validity, cri terion- related /alidity, 
predictive val idity and discriminant validity (American 
Psychological Assoc la t i on , 1 974 ) * 

Since the advent of per formanee-- based education, with its 
em phas Is on c r i tor ion-ref creneod mensu rem en t , organ \ ? t ing 



Defining Valid ity 




validation efforts around these types of validity has" proved 
difficult (Grant, Note 2; Messiek, 1980; Popham , 1978). When 
first faced with these issues in 1976, we attempted to simply 
modify the existing types of validity (content, face, construct, 
criterion- related, predictive) to fit the outcomes and assessment 
techniques we were validating. That attempt ^ailed. The 
purposes and characteristics of instruments have changed, and we 
now need assessment techniques designed to measure abilities 
which consider the role of assessment techniques and processes in 
the teaching/learning process, the need for demonstrating the 
equity of the instrument and the importance of giving feedback to 
students. Governance questions related to who decides on 
criteria and standards are also an issue. Often, we do not have 
a clear picture of the complex constructs we are trying to 
measure. They are often developmental constructs, and we expect 
change. Test/retest reliability is therefore not a goal. Nor do 
we expect that abilities developed in college will have a 
straight line prediction to how they are demonstrated after 
college or even how they are defined. We are interested in 
developing abilities. Prediction to success in_ college is not as 
important as having diagnostic information on which to build 
instructional practice. Other issues relate to effective 
approaches for establishing the validity of programs, assessment 
techniques and outcomes which focus on the need for evaluation as 
well as validation efforts, and which consider the contextual, 
developmental and illuminative nature of programs (Weiss, 1983). 
We soon came to realize that we needed to rethink validity based 
upon our new assumptions about its use and function. Faculty 
questions provided the framework for designing a validation model 
and creating validation strategies. The nature of the questions 
and their relationship to various aspects of the learning process 
model (competences, experiential learning, assessment process and 
techniques) will be discussed in the next section. 

Out of this experience, we have come to think of two type3 of 
validity, design- based validity and pe rformance- based validity. 
With design-based evaluation and validation strategies in place, 
the research results from per for man o e- based validation strategies 
/ire more likely to be incorporated into program development 
efforts. If a program is constantly changing and assessment 
techniques consistently revised, new information has n place to 
^o--a place to begin to be tested in the practical context fro fn 
which it arose. ^or each of the two types of validity, we later 
specify the nature of the questions asked by faculty, which 
determine comparisons against internal criteria or standards and 
those ox te rnrtl. to the prog rata, and how these comparisons will be 
offer: ted. 



Des i gn-Mased V'i I i d i ty 

!V?s i gn-hased validity has it,:; basis in criteria which faculty 
M.\n to define com|>» tenons , develop learning strato^ios, and 
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design an assessment process ( Figure 1). But as every curriculum 
designer knows, what looks good on paper needs adjustment and 
monitoring to make it work in actual practice. Program review 
and monitoring procedures are critical to establishing 
design-based validity. But how does one know a program is 
meeting these goals? Design- based vgtl idity refers to those 
strategies that monitor program function and compare the program 
against criteria or standards evoked during program design ("What 
is occurring? How is it occurring? What should be occurring?" ) . 
Both internal and external criteria or standards about how 
program components should be designed are U3ed to answer the 
question "What should be occurring?" both during design and 
implementation. ...For example, identifying competences, learning 
strategies and an assessment process evolved from the expert 
judgment of faculty (internal criteria or standards) who had vast 
experience teaching and ■ assessing students. Faculty drew on this 
experience to create the various components of the learning 
process. For example, one design criterion or standard for 
defining competence is that competences be defined as 
developmental, holistic and generic. A criterion or standard for 
developing assessment techniques is that the performance mode be 
similar to the ability as it is usually expressed. The corporate 
faculty pooled their resources as designers. Thus, one source of 
criteria or standards is the expert judgment of the faculty. 

Design-based validity does not necessarily rely entirely on 
faculty judgment based on their own criteria or standards. 
Criteria or standards from outside the college (external 
standards) are drawn from various sources. For example, 
professional groups were consulted on the definition of 
abilities. Expectations about the nature of the abilities needed 
in personal and professional roles of graduates (future outcomes) 
were also discussed. Literature reviews were also used. 

A. program can be said to have design-based validity when the 
comparison between what is intended and what is actually 
happening on a day-to-day basis at any one point in time is 
realized. This comparison is effected through a variety of 
review procedures carried out in relation to various aspects of 
the curriculum (e.g., Assessment Committee eva! mtes instruments; 
syllabi are submitted for review; external assessors from the 
MiLwaukee community judge student performance and critique the 
assessment process; Mentkowski, 1980). 

Performance-Based Vn 1 id ity 

design-based validity alone can be tautological. Even though 
designers and implementors consult outside resources, there in a 
ri^ed to measure program outcomes, in our case this means the 
performance of students. Performance- banod validity refers to 
the strategy of reviewing student performance of outcomes as it 
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develops through instruction rather than comparing how the 
program functions against internal and external criteria or 
standards ("What is occurring, how is it occurring, why is it 
occurring?"). Validity rests on whether student performance 
changes over time as the result o f instruction, and whether these 
changes persist beyond college. At the same time, one cannot 
3top there. "Is the change in student performance 'good 1 
compared to a standard?" is still an important question. 

Thus, changes in student performance need to be compared 
against criteria. For example, suppose that a study of student 
performance show3 change on a faculty designed measure of 
analysis. The faculty can ask, "How does the range of 
performance compare with how we have defined the ability 
(internal criterion or standard)?" They may also ask "How do 
students perform on an external criterion measure of analysis 
developed by this researcher of analytical thinking?" or "How 
would professionals in management demonstrate analysis, and do 
our students show the beginnings of this ability as expressed 
on-the-job?" (external criteria or standards). 

One of the first questions we must deal with in measuring 
outcomes and future outcomes is the identification and source of 
criteria to which outcomes will be compared. The basis for 
establishing validity is comparison. But what should comprise 
the nature of the comparison 9 Whose standards, and what kind of 
standards are adequate? The search for standards to which 
program and student outcomes car. be compared is a continuing one. 
In the section "Defining, Assessing and Validating Outcomes in 
Higher Education," we pointed to the lack of frameworks and 
measures available for use as standards to which a college's 
outcomes could be compared, We have chosen those frameworks and 
measures more likely to meet certain of our own criteria for 
outcomes and measurement techniques. Yet, we recognize that 
choice or selection of any criteria—whether a measure of 
cognitive development, a set of abilities that describe 
professional performance at work, a set of norms based on a range 
of student performance, advice from a group of external 
assessors, or goals from program designers— is somewhat 
arbitrary. We deal with this question by using a variety of 
approaches to establishing validity, using a variety of criteria 
or standards from sources both internal and external to the 
learning process and the performance of our students. 

However, faculty selec ti ng a standard should consider its 
representa tivenons or the extent to wh i oh the standard Ls 
inclusive of the interest group. Second, a standard should be 
valid. To what extent is the external standard meaningful? If 
both these questions cannot be answered to one's satisfaction, 
the external standard itself may need to bo validated before 
including it In a validation study. Thus, types of criteria or 
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standards include those developed by faculty as well as those 
identified outside the institution. 



In addition to focusing on questions about changes in student 
performance over time, performance-based validation strategies 
examine the relationship be twee n the program and student 
performance. This comparison allows us to determine the 
effectiveness of the curriculum, or rather, the interactions 
between the curriculum and change in student performance. This 
comparison is effected by observing changes in student 
performance over time in relation to educational experiences 
(instruction). The results of design-based validation studies 
are thus further challenged by performance-based validation 
strategies, just as the results of studies of changes in student 
performance are further challenged by comparison to external 
s tandards . 



Thus, the true test of a program ( design- based validity) is 
its relationship to changes in student performance over time 
(performance-based validity). And the true test of student 
performance is to examine how student performance changes over 
time in relation to educational experiences, and whether these 
changes persist beyond college. Ultimately, both the degree and 
type of change in student performance of outcomes over time is 
compared to internal and external standards. 

Making the shift from the traditional types of validity to 
design-based and performance-based validity helps to 
conceptualize validity given the assumptions that have been 
previously specified abotr. its role and function. Validity is 
developmental, a process, considers theo ry-prac tice, 

research-evaluation relatedness, and is contextual. Strategies 
for establishing performance- based validity are ongoing. In a 
continuously changing program, design-based validity is also 
ongoing. Re-design is often concurrent with attempts to 
establish performance- based validity. We cannot expect that a 
faculty carry out performance-based validation strategies on 
alumnae until there are graduates, nor can a faculty validate 
criteria for assessment until outcomes have been identified and 
defined. In general, however, attempts at performance-based 
validity will be simultaneously attempted with design-based 
validity strategies. 

In some ways this is an advantage. An stated earlier, with 
designed-based evaluation and validation strategies in place, the 
research results from performance-based validation strategies are 
more likely to be incorporated into program development efforts. 
If a program is constantly changing and assessment techniques 
consistently revised, new information has a place to go--a place 
to begin to be tested in the practical context from which it 
arose . 
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Identifying Va 1 i da t ion Qu est i o ns 

As stated previously, our assumptions about validation and 
our definition of validity arose from questions faculty began to 
ask as they designed, implemented and tested the curriculum 
against s tuden t perceptions and pe rf ormance . Tn an ongo ing 
curriculum these questions continue to be asked, since it is 
unlikely that a changing curriculum will ever be validated in an 
absolute sense, nor do we think it should be. 

Earl ier, we categorized ques tions into four general ki nds : 

• Descriptive questions: "What is occurring?" 
"How is it occurring?" 

• Ascriptive questions: "Why is it occuring?" 

• Evaluative questions: "Is what is occurring 
'good* compared to a criterion or standard?" 
"Is the standard valid?" 

• Prescriptive questions: "What should be 
occurr ing?" 

Establishing design-based and performance-based validity 
means applying these questions simultaneously to the curriculum 
components anT to student performance of current and future 
outcomes. Asking descriptive questions implies observation and 
measurement of changes in student performance over time. Asking 
ascriptive questions implies establishing relationships between 
various curr iculum components and current and future outcomes . 
Asking evaluative questions implies a comparison between 
curriculum components and student performance of outcomes, to 
internal and external standards, and asking if those standards 
are valid. Asking prescriptive questions implies implementing 
research findings to improve current understanding of student 
needs and curriculum practice. Becar.se the questions are applied 
in an ongoing and changing curriculum, there is a need for 
investigating all four type3 of questions simultaneously. In 
order to respond to these questions, faculty created an 
additional component of the curriculum In addition to 
competences, experiential learning and assessment process. This 
compo nen t is call ed evalua t ion/val idation process and techn Lques 
(Figure 2). In order to establish design-based validity, the 
faculty created internal review, evaluation and revision 
mechanisms at the same time a3 the program was designed. 



An Office of Research and Evaluation was created three years 
after program implementation to establish performance-based 
validity. Faculty questions that stimulated the more systematic 
pe rfo rinanee-banod validation research through the Office of 
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Research and Evaluation can be categorized with reference to the 
curriculum component against which it is applied (competences, 
experiential learning, assessment process, student outcomes , 
future outcomes), and whether the criterion or standard to which 
the outcome is compared is more likely to be internal or 
external. 



Validation of Competence Compared 
to Internal Criteria or Standards 

• Are our assumptions about the complex 
nature of each competence adequate? How 
best should the ability be defined so that 
its meaning is clear? Have all aspects of 

. the ability been defined? 

• Are the competence levels actually sequential? 
Is one c ompe tenc e level necessary in o rd er to 
demonstrate the next level? 

• Is each competence level more complex than 
the previous one? Dees the next level 
appear more complex only because it is 
integrated with more complex content? 

• Have all the significant relationships 
between the competences been identified? 

• Are aspects of an ability or competence 
common or generic to each discipline 
identified and measured? 

Validation of Competences Compared 
to External Criteria or Standards 

• What competences do professionals perceive 
as critical for outstanding performance, 
education and selection? 

• What competences do effective professionals perform? 

• How do professionals describe their careering and 
professiona L development? 



Faculty Questions for Establishing Valid i ty 
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Validation of Experiential 
Learning Compared to Internal 
Criteria or Standards 

• Do learning experiences reflect the basic 
tenets of experiential learning in both 
classroom and field experiences? 

Validation of Experiential 
Learning Compared to External 
Criteria or Standards 

• What gains or changes in performance do 
students demonstrate as a result of the 
learning pro cess? 

• To what aspects of the learning proce33 do 
students attribute their development? 

Validation' of the Assessment 
D rocess Compared to Internal 
Criteria or Standards 

• Are criteria used to judge performance 
in delation to the competences valid? 

• Is the instrument stimulus and mode of 
assessment appropriate? 

• Are the judgments of performance reliable? 

• Do assessment techniques measure the effects 
of instruction? 



Validation of the Assessment 
Process Compared to External 
Criteria or Standards 

• How does the assessment pro* ess compare 
to.; assessment center standards? 

Validation of Changes in Student 
Outcomes Compared to Internal 
Criteria or Standards 

• What is learning to learn? 

• How do students learn to learn? 

• How do students l^nrn to loam from 
experience? 
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• How do students learn to learn on the job? 



• What are the learning outcomes or processes 
each student is able to demonstrate? Are 
outcomes defined in ways that reflect what 
we' understand about students and the 
development of the abilities? 

• How do abilities or competences develop? 

• To what extent are abilities or competences 
developmental? Are they teachable? 

• To what extent are abilities or competences 
holistic? Are they internalized and 
characteristic of the person? 

• To what extent are abilities or competences 
generic? Do students generalize their 
performance across- time and situations? 

Validation of Change in Student 
Outcomes Compared to External 
Criteria or Standards 

• How do students change on college outcomes 
described by their potential — what is 
possible for them to achieve? 

• How do student outcomes compare with 
outcomes from students at other colleges? 

• How are outcomes we assess for mirrored in 
students' perceptions of their developing 
ab ilit ies? 

• How are outcomes, abilities or competences 
achieved in college causally related to 
effective performance of professionals 

a t wo rk? 



Validation of Future Outcomes 
Compared to Internal Criteria 
or Standards 

• What are the future outcomes, abilities 
or competences alumnae demonstrate in 
their professional performance? 




• How are alumnae outcomes we identify 
mirrored in their perceptions of their 
developing abilities? 

• How do alumnae transfer abilities to life 
after college? 

• How is lifelong learning characterized? 

• How do alumnae demonstrate careering and 
pro fessional development? 

• How do alumnae relate personal and 
. t , professional roles? 

Validation of Future Outcomes 
Compared, to External Criteria 
or Standards 

• How are outcomes learned in college 
related to ^ graduates 1 future personal 
and professional performance? 

• How do alumnae futuro outcomes compare to 
those demonstrated by outstanding professionals? 




RESEARCH OBJECTIVES AND REPORTS 



Questions faculty asked about the validity of college 
'tcomes were operational ized into research objectives within an 
overall validation model. The model included specific questions, 
designs, instruments, and procedures. so the more systematic 
validation of outcomes might occur. While we argued that faculty 
questions are researched simultaneously to a degree, we also 
realize that validation is developmental, and that it will be 
carried out within a particular context. We are, therefore, 
selective in carrying out a* program of research which may be 
directed to most but not all components of the validation model 
at a particular time. 

The assumptions and faculty questions /presented so" far in 
this report contribute directly to the research objectives 
specified in the prior reports to the National Institute of 
Education for the grant "Careering After pollege: Establishing 
the Validity of Abilities Learned in College for Later Success" 
(Mentkowski \ Doherty, 1977, 1979, 1980a, 1980b). These research 
objectives are: 

To internally validate the competences and assessment 
techniques of the Alverno learning process by-- 

I. Seeking to establish the validity of the 
techniques used to assess student 
performance by adapting or developing 
validation techniques appropriate for use 
with non- tradit ional assessment instruments; 
assessment instruments ; 

TI. (a) Comparing student performance across 

and within competences to further refine 
the nature of the competences and their 
inter- relat ionships ; 

(b) Examining the relationships between student 
performance and external criterion measures- 

To externally validate the student outcomes of the Alverno 
College experience by— 

III. (a) Comparing the competences identified by 

Alverno with the competences demonstrated 
by outstanding professionals; 

(b) Following the future careering of our , 
graduates in their various professions 
after college; 
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[V. (a) Generating in-depth profiles of student 
perceptions of themselves and their 
development and analysing the 
relationship of these perceptions to 
Alverno's learning process; 

(b) Assessing student attitudes toward the 
learning process; 

V. (a) Assessing students on cognitive-developmental- 
outcomes identified as descriptive of 
individuals who have reached various levels 
of potential in ego, moral, and intellectual 
developmen t; 

(b) Assessing students on generic competence 
external criterion measures that assess a 
variety of analytic and interpersonal 
abilities. 



The ten research reports that comprise the full report 
respond to the objectives as initially stated. The more specific 
questions that followed from these objectives have been stated 
earlier. They are formulated to best communicate results to the 
more general higher education audience. 

Therefore, the five questions listed below structure the 
complete report. 

• Are the com pete nces and A^ 3 A 3 J?!!lgILlL JL e j^jHLL^J^!. * 

of t he lear ning process valid? 5 
(Objectives I and II above!" 

• How do studenjg_j^aj^ 

described by their poten tial ^cg-J^g^L^l 
development , J earning _style_s and generic abilities? 
(Objective V above) 

• Are o ut_com es mirro red in studen ts' pe rcep tions 
of th e ir le arning _a n d a bilitie s? 

XOb jec ti ve IV abov ej 

• How do outcomes le arned in coll eg e relate 

to lifelong learn i r\ g, a bilitie s, car eerin g _and. 
professi onal ~~d ev efopm ent a fter co llege? 
(Objective III, b) 

• What competenc er, de scribe^ the performance and 
perceptTons ~oT out s^nj^ing A^°Jl^l^/i.l?. ? 
(Objective III, a) 
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Each of these five questions is related to one or several of 
the faculty questions listed previously. At the same tlmrs n M Hi 
represents a separate, involved research approach. The 
relationships between each of these approaches are apparent from 
the previous section. We will later provide links between the 
conclusions we draw from each of the five research thrusts, and 
describe implications for validating the outcomes of 
outcome-centered liberal arts curriculum. 

that the reader may relate these questions to the 
components of the validation model presented next (Figure 3), 
each of these five questions is listed again, with the questions 
that form the basis for each study. 



Question I , • Are the competences and assessment techniques 
of the learning , jp r ocess v al i d ? ~ 



• Do competences . reflec t our understanding 
of how they develop? Are competences 
developmental? 

• Do competences involve the whole person? 
Are competences holistic? 

• Do competences generalize across time 
and situations? Are competences generic? 

• Are assessment criteria valid? 

• Is assessor expert judgment reliable? 

• Do instruments measure the effects 
of instruction? 



Friedman, M. , Mentkowski, M., Earley, M. f Loacker, G., & 
Diez, M. Validating assessmen t techniques in an 
outcome-centered libe ral ar ts curr ic ulumT ValiTing 
and Communications G en eric I nstrument ! Milwaukee, WI: 
Alverno Productions, 1980." 

Friedman, M. , Mentkowski, M. , Deutsch, B. , Shovar, M. N. , # 
Allen, Z. t Validating asse ssment techn iques in an 
ou tcom,e-centered liberal arts curriculum" : Social 
Inter action Generic I nstrument . Milwaukee , wf: Alverno 
Productions, 1982. 



• Wha£ evaluation, revision and validation 
techniques are more appropriate for 
nontradit ional assessment techniques? 

• Which generic assessments are better 
indicators of college performance and 
performance characteristics that can serve 
as cross-disciplinary outcome measures? 




* How do students change on generic measures 
of student performance? 

Assessment .Committee/Of fice of Re.search and Evaluation. 
Validating asses3men_t techni ques in an o utcome-centered 

liberal ar ts cur riculu m: Ins ights from the 'evalu atio n 

and revision pr ocess . Milwaukee, WI: Alverno 
Productions, 1 980 . ^ 

Assessment Committee/Office- of Research. and Evaluation. 
Val id at j. ng assessmen t t echnigues i n an outcom e-centere d 
liberal arts c u rriculum: Integrated C ompete nee Seminar. 
Milwaukee, WI: Alverno Productions, 1982. ' 0 

Assessment Committee/Office of Research and Evaluation. 
Va 1 i d a t i ng asses s ment te c hniqu es' in an but come - cente re d 
liberal arts curricu lum: Si x Perfor mance Ch arac teristics 
Rating." Milwaukee , WI: Alverno Productions, 1983. . 

Assessment Committee/Office of Research and Evaluation. 
Six Per formanc e Cha racteristics Rating . Milwaukee f - WI* 
Alverno" Productions, 1978, Revised 1979. 

Question II • How do stud ents change o n colleg e outcome s 
"de scrib ed by "the ir potential for ,cogni_tive 
de v el o pmen t , 1 ea rni ng st yles and gene ric abilities? 

1 • How do students change over time on 

measures of human potential — cognitive 
development, learning. styles and generic 
/ abilities? 

t^Can change be attributed to performance in 
the learning process rather than to 
differences in age, background, or college 
program? ' * 

* What patterns of change emerge in the 
interrelationships of the human potential 
measures of cognitive development, learning 
styles and generic abilities, and generic 
measures of college performance? 

Mentkowski, M^, # Strait, M. A longi tu dinal study of 
stude nt ch ange in cogni tive de ve lopment an d generic 
abilities in an out come- cen te r ed li beral a rts 
curriculum. Milwaukee , WI: Alverno Productions , 198*5. 

Mentkowski, M., Moesier, M. , h Strait, M. Usin g the Perry 
sc heme of intelle ctu a 1 and eth ical J^^^PJJ^^J^Jl J* } 

coTlege outcomes rneasnrej A p_r_Qceg3_ a.nd c rj teria ^f or 

judgin g stud ent p erforma nce. V0I3, 1 & 2. 
Milwaukee 4 , WI: Alverno Productions, 1983. 
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Mentkowski, M. , Miller, N. , Davies, E. , Monroe, E. , * 
Popovic, Z. r Using th e Se ntence Completion _Test 
measuring Lo evinger 1 s sta ges of ego dev elopmen t a s a 
college outcom es mea sur e; K iting large numbers of 
- pr o t o c ols and m a i n taining v alidity o f the ra ting . 
Milwaukee, WI: Alverno Productions, 1981. 

Me n t k o ws k i , M . Change s in student pro fi_les on the le arning 
style inventory. First Report to Participants in a 
Longitudinal Study of College Outcomes. Milwaukee, WI: 
Alverno Productions, 1981. 

Mentkowski, M. Understanding t h e_ d ev el o pm en t o f think Lng 
in college* Second Report to Pqrtic i pants in a 
Longitudinal Study of College Outcomes. Milwaukee, WI: 
Alverno Productions, 1981. 

Mentkowski, M. Some questions and answers abou t ev aluation 
studies. Third Report to Participants in a Longitudinal 
Study of College Outcomes. Milwaukee, WI: Alverno 
Produc tions, 1 979 - 

Mentkowski, M. , % Fowler, D. Lear ni ng co learn at work ; 
Studen t s , alumnae and other pro fessionals. Fourth 
Report to Participants in a Longitudinal Study of 
College Outcomes - Milwaukee , WT: Alverno 
Productions, 1981 . 

Question ITT « Are outcome s mir rored in stude nts ' pe rceptions 
of their learning and abilities? 



• How do students understand and justify 
learning out comes? 

• How do students understand outcome-centered 
liberal learning as relevan t to performance in 
personal and professional roles? 

Much, N. , 8l Mentkowski, M. Student per spe c tive s on liberal 
learning at Alverno College: Justifying learning as 
relevant to performance in personal and pro fessional 
roles. Milwaukee , WI: Alverno Productions , 1982. 



Mentkowski, M. Alverno C o llege Attitu de Surv ey. Milwaukee, 
WI: Alverno Productions, 1977. 

Mentkowski, M. , h Much, N. Alvern o College Stu d ent 
Pei spec tives Inte r view . Milwaukee , WI: Alverno 
Productions, 1980. 

Mentkow3ki, M. , & Bishop, J. Alverno C olle ge S tudent 
Careering Questi onnair e. Milwaukee, WI: Alverno 
Productions, 1981 . 
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Question TV e How do out com 03 learned in vollet^j'Qrtj}_J:o_ 
TTfeTon^ learning, ^a_bi 'l_\t i^j^f careering and 
p r o f e s s i 0 n a 1 d ov el 0 pm en t after c 0 1 \e& e ? 

• How do expectations of students and 
re.a 1 i zat ion s o f a 1 umna e com pa r e? 

• What abilities and processes enable 
transfer of learning to professional 
performance and careering after college? 

• How are alumnae learning to learn at work, 
and do they describe lifelong learning? 

• What are alumnae perspectives on careering 
and profess tonal development? 

Mentkowski, M. , Much, N. , S Giencke-Uoll , L. Careering 
after coll ege : ?e rspec tives on li f elon g learning 
"career Tevelo pmen t . Milwaukee, WI: Mverno Productions, 

Mentkowski , M . , % Mu c h , N . Alv erno C ollege ^ljijnna 
?e rspec tives Interview, Milwaukee , WI: Uverno 
Product io n s 7T9807 

Mentkowski, M. , 4 Bishop, J. A 1 v e r n o Col lege Alumna 
Careering Questio nnaire, Milwaukee, Wi : Alverno 
P7oTucTtTo'ns , 1 9 80 . " "" 



• What perspectives and strategies do alumnae 
demonstrate in relating personal and 

pro fessional roles? 

Question V • Wha t competent e s_ _d_esc rib e_ the per formfflceja nrt_ 
perceptions of ou tstan din g pr o f e s s i o n a 1 3 ? 

• What competences do outstanding 
professionals in nursing and management 
perform? 

• What competences do professionals in 
nursing and management perceive a3 relevant 
to performance, critical for education and 
selection, and descriptive of outstanding 
performers? 

• Flow do professionals describe their careering 
and professional development and what aspects 
are related to performance? 
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Mentkowski, M. , DeBack, V., Bishop, J., Allen, Z. , -4 

Blanton, B. D eveloping a pr ofess ional competence model 
fo r nursin g ed ucatio n. Milwaukee , WI: AlveTrTo 
Product ionsV 1980. 

Mentkowski, M. , O'Brien, K . , McEaohern, W. f S Fowler, D. 
Developing a pro fessi onal com petence model for 
man a gement e duca tion . M i 1 wn u k e e , W f: /Tfv e"rnrJ 
Productions,"" 1982. 

Mentkowski, M. , O'Brien, K . , McEachern, W., % 

Fowler, D . Developi ng a pro fees iona 1 c o m_pe t enc e 
model fo r man agement educati on. FiVaT ~Re por* f"Sul mm ary 
for Participants. Milwaukee, WI: Alverno Productions, 
MB'). 

Bishop, J., Mentkowski, M. , O'Brien, K. , Birney, R. , 
Davies, E. , Pi McEachern, W. Management Perfo rraance 
Charact e ristics Inventory. Milwaukee, WI: "Alverno 
Produc tions, 1980. 

Mentkowski, M. , h Bishop, J. Manag ement Ca reeri ng 

Ques t ionna ire. Milwaukee, WI: Alverno p roducTions , 
1980". 

Alverno College Office of Research and Evaluation. 
Behavioral Event In teview Wr iteup . Milwaukee^ WI: 
Alverno Productions, 1980. 
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RESEARCH METHODOLOGY 



Sample 

The sample consisted of over 750 women students ages 17-55 at 
•entrance and over 60 two-year alumnae at Alverno College. Over 
^0 women nurses and over 100 women managers and executives ages 
25-66 from the Milwaukee community comprised the sample for the 
professional stud ies. 

Characteristics of the Validation Model 



Correlational Rather Than Experimental Designs 

Program development is multifaceted. Therefore, so is a 
validation design. We use multiple approaches, and demonstrate 
validity through establishing rela te dness , rather than by 
establishing cause and effect relationships. Because the 
outcomes are developmental and the curriculum is changing, we 
must use co relational rather than experimental designs. If one 
cannot design laboratory studies that will establish cause and 
effect relationships then one must capitalize on correlational 
relationships, and that demands a model where the questions asked 
are in relationship to each other. The findings from one set of 
questions have implications for another. 

For now, we have abandoned most experimental designs and 
methods for establshing validity. The emphasis is on comparison 
of changes in student performance over time against internal and 
external standards. We are not likely to use group comparison 
designs where one group consists of Alverno students, and another 
consists of students at a college which attracts students of 
similar demographics but does not have a performance-based 
curriculum. We have found that we cannot make accurate enough 
assumptions about where Alverno students and those from another 
college would be similar or different. Thus, the adequacy of 
such comparisons for providing accurate and useful results is 
highly questionable. And selecting a control college is 
impractical. We cannot really "prove" whether a constantly 
changing and evolving curriculum is effective or ineffective by 
using such experimental models. By comparing our students 
against external standards, however, we may have some indication 
of how our students compare to students at those colleges where 
similar instruments are used (e.g., Winter, McClelland # Stewart, 
1981), and a range of student groups contribute to general izable 
"norms." Clearly, all standards of this type arise partly from 
normative data. 

In add it ion, all students complete at least four levels of 
the learning sequence. There are no intra-institutional control 
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groups. In addition, faculty who may not explicitly teach 
students a particular ability are aware : it and may still teach 
it implicitly. We have begun to internally compare student; who 
complete four levels of an ability with those who go on to levels 
5 and 6 a3 part of their major field, V 1 the currently available 
c riterion measures , for the most part, -..onsure only small na rts 
of the complex abilities demonstrated at level n. Com} inr 
alumnae who graduated prior to implementation of the 
performance-based curriculum with more recent alumnae is also 
unwise. The new curriculum had too many of its roots in the old; 
particularly in some academic departments. And the effects of 
the women's movement on careering outcomes could hardly be 
separated from effects of the new curriculum. A developmental 
framework cautions us that abilities learned in college may not 
he visible in the same form in later years. The predictive 
validity of an ability may be difficult to establish if we look 
for "more of the same" in a follow-up study of graduates, rather 
than evidence that an ability is developing and achieving 
integration with other abilities. 

How Alverno students as a group compare normatively to 
students at other colleges receives less emphasis than how our 
students' individual gains over four years compare to (1) 
developmental norms, and other standards derived from the 
faculty's understanding of the abilities they teach toward, (2) 
students' perceptions of their own growth, and (3) standards 
drawn from external criterion instruments that most nearly 
approximate the m .a fiuremen t of the abilities that we teach 
towa rd . 



But we have built several characteristics into the model that 
allow us to move beyond some limitations in correlational 
designs. First, we employ aggregate, triangulated designs. 
Second, we U3e both longitudinal and cro33-3ec tional approaches 
that compare age and class cohorts. We use a time series design 
to attribute change to college performance, and match comparison 
groups for persistence in college. The3e characteristics of the 
validation model and longitudinal design are discussed below. 

An Aggregate Triangulated Model 

As far as possible, we approach every outcome or factor we 
study from several directions. Creating designs which ask 
questions simultaneously and focus on relatednes3 result in 
circularity of results and require that we use multiple sources 
of standards and study the development of multiple outcomes. We 
employ triangulat ion, which means that we measure the development 
of multiple outcomes and avail ourselves of multiple 
opportunities to compare student performance against multiple 
standards. As stated earlier, a changing curriculum does not 
al low for us ing experimental designs to research ascriptive 
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quest ions. Than, we rely on agg rebate findings; If we are able 
to demonstrate results in an aggregate, or variety of ways, we 
will have more confidence that our observations are true and 
repl icable. 



One of the values o f using aggregate findings and 
triangul nt i is that most questions related to the validity of 
programs ? , being asked somewhere in the model. While research 
takes time and effort, especially longitudinal research, some 
results with respect to a particular question are usually 
available. Since the research is carried out by in-house staff, 
they are aware of most sources of data and what is currently 
known regarding a part'cular issue. 



Here is an example of approaching an outcome from several 
directions. Consider the complexity of the process faculty have 
in mind when they use a phrase like "analytic thinking." 
Clearly, no single measure — whether devised by a faculty member 
for instruction and assessment or by a team of psychomet ricians 
for research — can hope to capture the whole of such an activity. 

So we aggregate several measures of different kinds, each of 
which bears upon some portion of the domain "analytic thinking," 
and takes several approaches at once. We can thus develop a 
general sense of whether something in that domain is changing, 
and can begin to ask questions about what it might be and why it 
is changing based on the differential responses the varied 
measures yield. 

Using tr iangulation helps solve other design problems. As 
stated earlier, experimental designs utilizing conrol groups are 
inappropriate in a changing curriculum. Ml students experience 
the "treatment" and even past alumnae, students from other 
colleges or persons not in college do not meet the criteria for 
serving as controls. In this manner we may pool successive 
results on in-class assessments from several different 
disciplines, results from several widely available measures of 
cognitive development and/or analytic reasoning (human potential 
measures), and results on student perspectives from the sequence 
of open-ended interviews. With this approach, we avail ourselves 
of at least three independent sources that are researching 
similar general questions. We can add to our understanding of 
more specific questions, while recognizing that results must be 
confirmed from other independent sources. We see, from several 
angles, phenomena we know are difficult to research given the 
practical limitations imposed by real-life ratfher than laboratory 
settings . 
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The Validation Model 



Figure 5 presents the validation model. It diagrams the 
three major approaches to validating outcomes in relation; to each 
other (performance, potential, .perceptions) to illustrate 
related ness and triangulation. The fourth approach utilizes not 
student performance and perceptions, but that of professionals. 

The sources of data have been identified and placed in 
relationship to each other in the design (Figure 3) in order to 
better describe the opportunities for rel atedn ess and 
triangulation. It is apparent that if questions are studied 
simultaneously, there are many opportunities for the outcomes 
defined to be further described and elaborated a3 the studies 
continue. 

Establishing relationships between changes in outcomes during 
college and future outcomes, is a complex task. It seems clear 
to us that the measurement of 3uch complex outcomes, and the 
measurement of change will proceed with many difficulties. There 
Is no simple one- on- one match between any of the outcomes, 10 r 
between ^outcomes and future outcomes. Clearly, the several 
approaches attempted simultaneously in Figure "3, while reflecting 
the complexity of questions asked, also demand a sophistication 
of strategies and instruments that we have not yet achieved in 
higher education. 

Since our purpose is to develop a validation design that is a 
proce33, we can proceed with our work in spite of the pressure 
that comes from researching evaluative questions, and 
demonstrating the "worthwhilene33 of the program and gain3 in 
performance. Indeed, given the state of the art in measuring the 
complex outcomes of college, we can make progress in some cases 
by describing those outcomes initially, and later asking 
questions of evaluation. 

Such a complex design can seem overwhelming. But we have 
found it helpful to specify our questions and some ways in which 
we can begin to a3k them. Asking the questions, rather than 
putting them off until adequate designs, strategies or 
instruments are available, seems to us a better way to grapple 
with their complexities. The design is an opportunity not to 
generate perfect "results," but to enable us to ask better 
questions - 

From the research questions, we have determined the bas Ic 
structure of a model for validating the curriculum and changes in 
student performance. It is clear that our task is to identify 
and measure changes in student potential. We must obtain a 
description of changes in student potential over time in relation 
to 3tudent achievement of competence in the learning process, to 
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establish the ivl it ionshi p between student potential and 
performance in the learning process. In addition, we must 
identify and measure future out-ones of graduates, and identify 
relationships between student and alumna outcomes. We must aLso 
identify and assess student perc ept [ions, since student 
perceptions are external standards against which performance and 
potential can be compared. We must also identify and assess the 
performance and perceptions of professionals. 



Cha ra cteristi cs o f__He sea r c h De s i g n s 

Longitudinal jand Cross-Sectional Approaches 

Rather than equate change with average gains, faculty are 
Interested in the extent to whish each student changes. ' The 
curriculum may facilitate growth only for students who are 
verbally skilled. Or students who enter college with already 
sophisticated abilities may coast through a port.ion of curriculum 
and make few, if any, gains. For facilitating individual 
patterns of change and growth, faculty designed the curriculum to 
include consecutive assessments throughout a student's college 
career. Consequently we are likely to select similar 
longitudinal designs. While longitudinal studies using external 
criterion measures are time-consuming and costly, they yield 
individual growth patterns. The following diagram provides a 
picture of the combined longitudinal and cross-sectional design 
used in our study of student and alumnae outcomes. Each dotted 
line represents one of three student groups assessed repeatedly 
in a time series. Each dot on the line represents an assessment. 
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Total Sam pi in/ 5 ; 

Students drop out of college and new ones enter in midstream. 
To enable longitudinal research with adequate sample sizes, we 
use total, sampling involving all students entering or graduating 
in a particular year, rather than random sampling. 

Afi!' fln d Age Co hort 

Prominent in our report is our concern with age and age 
cohort differences. Because our student population ranges in age 
from 17 to 55 years, and because we expect to continue to attract 
older students in the future, we have a special opportunity to 
examine change across a larger range of adult life. We have used 
age, broken down into traditional and older student cohorts, to 
compare the general influence of life experience, or 
"maturation," to formal education experience. We have also used 
age, standing again for life experience in general, as the 
logical first cause of differences in development and other 
abilities when examining the causes of change. 

Clas s Cohort 

For the purposes of general program validation, we undertook 
the extra effort of studying two successive years of class 
cohorts to minimize the possibility of unseen cohort effects in 
our general conclusions about change (Nesselroade <3 Baltes, 
1 979 ) * The cohort variable is not interesting in itself, but it 
proxies for whatever events on a social level were influential in 
student selection of a year to enter college. The age range of 
our population and the volatile environment of the seventies and 
eighties in the changing roles of women, make this issue 
particularly important for our women students, many of whom are 
first generation college students. 

Time 5 e r i e s Design 

It is part of our language to speak of "the four years of 
college" as if all the students who enter in a given year (or at 
least all the persisters) complete the program and graduate four 
years later. But this has never been the case. Indeed, with the 
influx of "new" students and their multiple life commitments, the 
four year model is already for many institutions a minority 
pattern rather than a norm. 

This i3 an important advantage for both our longitudinal and 
cross-sectional studies. We administer our external measures at 
entrance, two years later, and again a year and a half later, 
rather than when a student is a freshman, beginning junior or 
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graduating senior. Thus, the time at which students ore assessed 
on external measures is held constant, while the number of 
semesters they take to make that progress can vary. 

Because Alverno students are awarded credit for successful 
demonstration of their abilities, at successive levels of 
sophistication, we use two measures of progress. One is the 
qualitative accumulation of her demonstrated ability levels; the 
other is the more quantitative record of semester hours 
completed. Both vary because the number of ability assessments 
offered by an instructor, as well as the number attempted and 
completed by each student, is different in very course. 

This time-variant approach allows us to use the variation 
among students in the time they spend in college, and their 
quantitative and qualitative progress through the program as a 
basis for comparison. We can then explore such key questions as 
whether students who perform more successfully in this program 
also show more change on measures from outside the college. 

Thus, in our longitudinal design,, we assess students at 
consistent time intervals (see Table 1). Performance in the 
curriculum varies. Thus, we can investigate changes as a 
function of performance in the curriculum. Many studies of 
college effects, whether cross-sectional or longitudinal, have 
assessed students when they are Freshmen and when they are 
Seniors. In contrast, we have assessed an entire entering class 
as they began their studies, and then have reassessed the same 
group two years later, and for a third time, about two years 
later. Most entering students will be new Freshmen, but many 
will have prior college credits and, in cia3s terms, will be 
Sophomores or Juniors at entrance assessment. A typical student 
who entered a3 a nevr freshman and attended regularly for two 
years might in fact be a first semester junior at second 
assessment, but another 3tudent might have entered Alverno as a 
sophomore by standing, taken only cwo courses in the entering 
semester, not regis te red again unt il second asses smen t , and still 
be a sophomore. Class standing may be different at the third 
assessment two years later as well. Tt is precisely the 
variability in attendance and performance over a specified period 
of time that we use to investigate claims of change effects for 
the learning process as a global entity. Vlhen appropriate, we do 
take advantage of the fact that our design approximates the 
beginning, middle, and end of a typical student's college career, 
or that the assessment intervals approximate the periods of 
general : education and pre-prof essional education for the typical 
student. 
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Design for the Administration of Human Potential Measures and Student Percepti 
Measures for Longitudinal and Cross-Sectional Studies of Student Outcomes 
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Note. See Figure k for overview of components of the program validation model with measures. 
Student Perspectives Interview (SPI) data were collected on a subsample of students - 
participating in the administration of the Human Potential Measures (HPM) , but all completed 
the Attitude Survey (AS) and Careering Questionnaire (CQ). All Weekday College students completed 
the Integrated Competence Seminar (ICS) and were rated by faculty on the Six Performance 

Characteristics (SPC). 



Achievement Cohort 



In ^ performance-based curriculum, students must demonstrate 
successive mastery of the* competence levels. Their record of 
performance in the curriculum is an indicator of level of 
achievement in the curriculum a,t any point in time. Students 
vary in the number of competence units they have achieved, and 
can thus be categorized as high versus low achievement. The 
effects of performance in the curriculum can be studied in this 
comparison. 

hatching C om pa rison 0 r o u p s 
for Decree Completion 

A well known problem with comparing groups of entering and 
graduating students in cross-sectional studies is that entering 
classes include many students who will not persist through 
college, while a graduating group consists .of persisters by 
definition. Many studies try to dontrol for this difference by 
matching students on some variable believed to predict 
persistence, most often an academic achievement variable. 

We were able to, control directly for persistence in the 
c ross-3ec tional study by using one of the entrance cohorts of the 
longitudinal study who were in college for years later, as our 
entering student comparison group. Thus, our cross-sectional 
comparison provides a conservative estimate of change. 

Controlling for Other Factors 
That Contribute to Change in 
Performance 

To relate change to performance' in the learning process r a 
series of analyses were first conducted to control for other 
factors that contribute to change in performance before testing 
for performance effects . First , age and other background 
variables (religion, parent's education and occupation, high 
school grades, prior college experience, and marital status) were 
tested for correlation with entrance assessments. Those 
variables that accounted for differences in-entrance assessments 
were then further examined for relationship to change in 
performance between assessments. If any background variable 
accounted for change between assessments, then that difference 
was controlled in testing effects of performance. Similarly, 
effects of program differences incidental to the learning process 
(entrance cohort, residence, parttime or . fulltime status, "and 
major) were tested after background variables but before testing 
for performance effects. Program variables accounting for change 
o^er that accounted for by background variables were also 
controlled before testing the relationship of change to 
performance. Thus, any relationship between "performance and 
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change was only considered an effect of performance once the 
other possible sources of variance were controlled. 

Increasing Rate of Participation 

Renders experienced in longitudinal data collection will be 
interested in the extensive and detailed description we provide 
of our data collection procedures (Mentkowski % Strait, 198*5). 
We employed a range of effective strategies to get and keep the 
cooperation and participation of all students entering during the 
two-year entrance phase of the project. The continuous effort to 
motivate students to participate may be seen as reducing 
generalizability to populations not so motivated. We have taken 
the position that complete data is a more important goal. We 
have motivated students to participate in order to achieve the 
highest possible rates of participation, and to stimulate the 
highest level of performance. 

Tn addition, we employed a variety of strategies to insure 
the participation of professionals, particularly for those in 
management. These procedures are described in detail in 
M>>ntko wski , ft'rJrien, McFhohern a Fowler (19ft*?). 

Procedures 

Procedures for carrying out the research objectives were 
d es i g n e d to m ee t four b r o a d o b j ec t i v es : 

• Create a context for validation research 

0 F. e s pon d to concerns of s t ud e * 3 , fa cully, 
and pro f e ss io na 1 s 

• Collaborate with colleagues in research 
aril curriculum development 

• ''^vip^et trie values and objectives of the 
pro ram ! ind the renea rch p ir t i o [ pan ts 

M, o carry out these objectives, we devised two strategies, 
'he'* was to es f.abl ish evaluation as a concept and function at 
\ 1 v i m 4 n» ; 0 o j l f v ' t . » (figure ^ ) n nd t; h e s < * oo nd wa s t o d ev el o p rn j r o 
sp"r ifi.t- research methods thai, mirrored the value-,, objectives, 
'mi as sessinen 1; process -and teehni |uea of the college, as well as 
Mi*- recent frameworks and ins I, rumen t*i t i on in fields related 

V i * he .at 'jily •)!' college (hi' •:rji!w , :i , 
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Create a Context for Validation Research 



We created a context for validation research primarily by 
establishing evaluation an a concept and function in the 
curriculum. \s stated earlier, our colleagues in higher 
education were often interested in beginning 

evaluation/validation efforts of their own. For this reason, we 
document the context for evaluation that we established at 
Alverno, as a case study that occurred at one institution. For 
us, thi3 meant identifying evaluation goals for an Office of 
Research and Evaluation that could coordinate the functions 
necessary to carry out the research methodology and dissemination 
of the results. Consequently, this Office was created with the 
following overall goals. 

• Establish evaluation as a concept and function 

• Evaluate the quality, effectiveness and validity 
of the learning process 

• Con t r i but e to c r en ting a mo re general izab 1 e mod el 
of adult development and learning 

• Contribute to prog ram 1 evel o pmen t and s t uden t 
1 evel opmen t 

• *\ssist in insuring the quality of various 
research and ev al ua t ion ac tiv i t ies within the 
col lege 

• Establish Alverno an an accountable educational 
institution in the community and as a contributor 
to higher education research and evaluation 



The Office of Research and Evaluation was established in 1976 
at the beginning of our efforts t -> validate the curriculum, and 
is now a fully institutionalised and funded part of the college. 
The Office or' Research and Evaluation was created as a service to 
the college. The oval ua t ion/ val i dat ion process is a curriculum 
..ornpormnt (Figure that enables faculty to step back from the 
program objectively an 1 systemati rally, und take a hard look at 
progr-in functioning and validity in terms of student and alunnae 
ou tcom ■»:;. 



Respond to Concerns of Students, 
Kae u ! ty and p rof ess t on a 1 a 

[f research is to yie.d results with the broadest possible 
implications, not only for- the college, but also for th»- 
i rviiate ooirim mi ty it serv»»n, research personnel must consider 
Miems^l v*>s La service to the. broader goals, of their students and 
f aeul ty oo 1 1 eagu' i s . 
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J^?_cu 1 ty _J nvol vem e n t 

The involvement of faculty is critical to the identification 
of the research\questions, the carrying out of the studies and 
the critique and implementation of the results. Faculty were 
well aware of the concerns of higher education for validating 
outcomes, and the kinds of specific issues and questions that 
were central to curriculum reform. 



Several faculty groups were essential, to the validation of 
competences and assessment techniques. The Assessment Committee, 
consisting of members of the faculty who are specialists in 
assessment design, played a major role in designing and carrying 
out validation of assessment techniques. The Committee worked 
with the Office of Research and Evaluation to validate two 
interdisciplinary measures of college performance, and to create 
an instrument evaluation and revision process (Assessment 4 ^ 
Committee/Office of Research and Evaluation, 1980, 1982, 1983). 
The Assessment Center provided structure for administering and 
scoring of the Integrated Competence Seminar using external 
assessors from the business and professional community. The 
Assessment Council created definitions of the Six Performance 
Characteristics and in collaboration with the Office of Research 
and Evaluation, conducted the faculty rating of students on a 
measure (Six Performance Characteristics Rating) designed in the 
Office of Research and Evaluation to provide an external, 
cross-disciplinary measure of college performance of the broad 
outcomes of college. The competence divisions, most notably 
Communications, Valuing and Social Interaction worked as research 
teams to validate generic instruments (Friedman, Mentkowski, 
Deutsch, Shovar Allen, 19B2; Friedman, Mentkowski, Earley, 

Loacker & Dies, 1980). 



The Discipline Divisions, Chairpersons, Department 
Coordinators and all faculty members were involved in planning 
nnd carrying out strategies for involving students, for 
supporting the validation efforts, for communicating a rationale 
for the studios to individual students or classes, for 
identifying ways to approach and involve students in followup 
efforts to enhance the number of students participating, qnd for 
providing opportunities for Office of Research and Evaluation 
presentations. These presentations to students were designed to 
motivate students to pa r t i c i pa te , to give students feedback or, 
their performance on one of the measures assessing human 
potential, or to give students feedback on the overall 
<-'val ua t ton ^val idat ion results. The Dean's Office and the 
Assessment Center col 1 abo rated to plan the administration of the 
human potential measures so that involvement and participation 
ofru rred as part, o f regu] hp 'issessm^n t procedures. 
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Further, the Department of Business and Management and the 
Division of Nursing collaborated with the Office of Research and 
Evaluation to carry out the studies of professional perceptions 
and performance in nursing and management, and used their 
credibility and networking in the community to establish contacts 
with individuals and organisations. Several offices in the 
college, having direct access to the business and professional 
community, contributed information and contacts for the studies 
of professional competence: the Office of Career Development, 
the Office of Off-Campus Experiential Learning, the Development 
Office, and the President's Office. 

Student In vol v e m en t 

We were also responsive to students 1 concerns and ideas. We 
surveyed attitudes from half the student body in the spring of 
1077, and conducted indepth interviews of the first ten graduates 
In 1076 (Mentko wski , 1977b). This information was particularly 
helpful in focus Lng some of the research questions in our study 
of student perceptions. Issues identified for students included 
a focus on the relevance of education to careering after college, 
the Importance of improving the curriculum for all students, 
concerns with the validity of a newly formed program, Interest La 
performing well after college, and so on. Student participants 
in the research often asked questions that helped us to clarify 
and focus the questions wo were raising. These students also 
identified central concerns they had about participating so that 
adjustments could be made in data collection strategies. 

Involvement^ of Profess LjW^l_s 

Members of the business and professional community were 
already involved in the definition of competence (Advisory 
Councils), the creation and carrying out of experiential learning 
(through internships mentored by professionals in the field), and 
in the asses am e n t p ro o ess ( th ro u gh a s s e s sm e n t designers in 
business, and external assessors of studen*. performance drawn 
from the business and professional community). 

Kvulua t ion/ val idat ion efforts relied on such external input. 

Coll abo ra he wi th Co 1. 1 eagues i n Rosea rch 
and Curriculum Development 

\t the start of the NIK grant period, we were already 
collaborat ing with colleague in higher •education research and 
curriculum development. M verno College had just completed a 
broad dissemination of the learning process through grants from 
tun Kellogg Foundation and tin? Fund for the hnprovoinon t of Pont 
: .oeor.dary Education, and many colleagues from other institutions 
hud visited the campus over a period of years- We also 
main ta i an 1 con t inui ag ml at. i onsh i pa w? L a i ns t rumen t des igne rs 
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vsee "Instruments") and centers that were working toward issues 
similar to those in which we were involved, such as the Center 
for Moral Education at Harvard University, McBer and Company of 
Boston, the Center for the Application of Developmental 
Instruction at the University of Maryland, and Loevinger's 
research team at Washington University in St. Louis. 



Early on, as a member of a consortium coordinated by McBer 
and Company and funded by FIP3E, we helped test new measures of 
college outcomes. We worked with members of American College 
Testing and the Educational Testing Service, who were interested 
in developing innovative measures of college outcomes. At the 
same time, we were drawn to members of the research community who 
were experimenting with new measures of competence, and 
strategies for assessing competence, as well as theorists in 
cognitive development and their colleagues who were measuring 
patterns in human cognitive growth. We identified an Evaluation 
Advisory Council made up of experts from other institutions who 
could provide more specific technical assistance. Our Evaluation 
Advisory Council (Donald Grant, University of Georgia; Milton 
Hake I, Ohio 'Uate University; Joel Moses, AWr) assisted us in 
many issues related to design, instrumentation and validation 
during several visits to the c-mpus. 



A major contribution was made through extensive discussions 
on the issues conducted by Jean Miller of the National Institute 
of Education. These discussion meetings involved directors of 
five other projects. These sources, together with experts from 
our own faculty, formulated the more specific questions. Alverno 
faculty were part of the research teams. This was important 
because they would be primarily involved in tryout, 
implementation and dissemination of results. The overall process 
of insuring responsiveness to the work by students « nd faculty 
contributed directly to ttuj quality of the work. Thus, the 
methodology was constantly under critique and review. 



Hospeet the Values and Goals 
of the T'rograrn 



'Jj 5 o E x i s t l ng P r o g r a rn 
Eva 1 u a t i o n i > t ru o t u re s 

Sinc^ evaluation/validation is a program component (Figure 
.'), it noecossnrily follows that the methodology it carries out 
aeeds to be consistent with the objectives and methods of the 
other program components. Values underlying methodology need t<- 
be consistent as well. For us, several informal, nonsys temat i c 
processes for program revision and ^valuation were already built 
into 
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into the program at the time the validation research was begun. 
For example, faculty understand very well what student 
perceptions and attitudes toward the curriculum are. In a 1977 
study of student attitudes where half the student body completed 
a survey (Mentkowski, 1977a), all faculty individually completed 
the 3urvey the way they thought students would. Faculty 
accurately predicted modal student attitudes toward the program, 
toward faculty, ?.nd toward educational and administrative 
services (Mentkowski, 1977b). In this case, we tested the 
informal network for evaluating student attitudes in a more 
systematic way and demonstrated its effectiveness. This informal 
evaluative network is critical to planning strategies for 
involving student participants in validation research. 



We relied on systematic, in-place evaluation and revision 
precesses to carry out th& research objectives. An example of 
these processes is the one established to insure design-based 
validity of the program. It includes includes regular review, 
evaluation and revision of assessment techniques established by 
the Assessment Committee for the faculty. These reviews figured 
heavily in u he design of strategies for validating assessment 
techniques. Competence and Discipline Divisions also play a role 
in program evaluation and members of these groups served on the 
research teams for several of the studies. 



Prov ide Fee dback for 
Program Improvement 

In an educational environment, validation is more a manner of 
making incremental and qualitative judgments than of making 
static and quantitative ones. It is interesting, after all, to 
know that students during a five year period demonstrated certain 
patterns of development and that these aeem attributable to 
certain elements in the experience of "the college." But by the 
time we can make that statement, it is old news-. 



What evervone really wants to know is whether today's or 
tomorrow's version of "the college" is likely to have similar 
impacts on today's oi' tomorrow's students. Validation studies, 
properly designed, can help. They can enable us to make 
incremental judgments about whether and how the college is 
maintaining ' and improving its effectiveness in delivering 
desirable "outcomes , as it evolves to meet the needs of subsequent 
cohorts and moves into new program areas. 

Nor does val idation simply stand aside and judge the 
college's evolution; it contributes directly to the faculty's 
?| tfcom p t; , t,o improve programs. The trad i t iona! concept o f 
objective detachment La impossible from the outset, since the 
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faculty a active collaboration is needed in defining and 
redefining outcomes, as well as Ln devising at least some of the 
means and arranging most of the opportunities) for measuring 
student attainment of them. 



Attempting later in the process to "protect" the program or 
study participants from the impact of validation results would 
clearly be unethical, since qualitative feedback toward improved 
performance is the prime motive for both faculty and student 
participation. Tt would also be self-defeating. Closing off 
dialogue with the practitioners would immediately undermine the 
study s own validity, since it would cut off the primary source 
tor making adaptive changes in validation methods and strategies. 
It would also lead, in practical terms, to disaffection and rapid 
termination. ' 



Instead of mutual detachment, the operating mode in 
validating a dynamic educational program is mutual collaboration. 
Joining with the validation team to interpret even early results 
faculty then apply what they have learned and attempt to improve 
their program. The past becomes, in effect, the control and the 
present is an experiment in incremental change. If program 
modifications in turn yield improved outcomes, then 'the 
validation effort is itself val idated along with the faculty's 
efforts. In a constant dialogue characterized by ongoing 
feedback and collaboration, practitioner and validator thus help 
each other to sharpen their focus, deepen their understanding, 
and improve their effectiveness. At certain points, our methods 
and results can be set forth for review by our Advisory Councils 
made up of experts in validation who serve as other, more 
external sources of critique and input. 



Respect the Values and Goals 
of Research Participants 

Contacting _?_a_r tiej^pan ts 

A central concern in involving all participants in the study 
wis to ensure that contacts with participants and organizations 
met standarls for involvement, including informed consent 
confidentiality and feedback on the results of the studies as 
they became available. We were conducting longitudinal research 
with students and alumnae. Ineffective procedures could doom the 
project from the start. Further, some of the research was 
designed to build a bridge between the college and alumnae, and 
between the college and the professional community it serves. 
Our contact, procedures, by communicating our efforts, could be 
expected to positively contribute to the reputation of the 
college :\ n ] tnn degrees it offers. 



Wo consulted various members of the faculty and students in 
identifying strategies for contacting students. We also involved 
members of the college, Boari of Trustees, Advisory Councils, and 
members of the business and professional community to identify 
the most appropriate ways to contact professionals in nursing and' 
management. We intended that procedures would respect 
professionals' right to informed consent and that would respect 
the protocol, administrative structures and values of the 
institutions and organisations who employed them. 



Comm on i ca ting Rationale 
anc * Conf iden t ial ity 

In consulting with faculty and students, we identified the 
best ways to : inform students who were research participants 
(Mentkowski 4 Strait, 1985). Prior to each assessment in the 
longitudinal and cross-sectional studies, we made presentations 
on a regular basis to the faculty, informing them of upcoming 
student involvement, ways it would impact their class schedules, 
and the rationale for the studies so that they would he able to 
respond to student questions about the nature of their 
involvement. We consistently made presentations to the students 
in classes about the rationale for their participation. Students 
were contacted individually if group presentations were not 
workable. In order to maintain student confidentiality of 
participation in the interviews of student perceptions which 
involved a subsample of students, all contacting was completed by 
private letter or by research staff who maintained 
confidentiality. When whole classes of students were 
involved — and who was involved was public knowledge— facul ty were 
consulted about our procedures and their affect on individual 
students, who for personal reasons, were not participating. 
Great care was taken to involve student participants in the 
rationale for the study. On rare occasions when a student 
refused participation, such refusal was of course- respected. 
Confidentiality of individual performance was maintained 
throughout the studies through a system of code numbers to which 
only research staff had access. 



Student input in developing procedures for contacting and 
involving students in the work w^s particularly helpful (see: 
"Questions and Answers about Evaluation Studies: Third Report to 
Participants in a Longitudinal Study of College Outcomes," 
Mentkowski, 197'-)). Students critiqued study procedures and 
offered alternative suggestions that would assist in developing 
more effective strategies. 



We also took care in the i r. /ol vern^n t , informed consent and 
. oaf i lential Uy of professional participants. Organisations and 
their executives were contacted initially with attention to 
rationale and informed consent. Participants were contacted by 
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executives or administrators, and by the researchers with a 
rationale asking for participant consent. Confidentiality was 
promised and carried out ( Mentko wski , DeBack, Bi3hop, Allen & 
Blanton, 1980; Mentkowski, O'Brien, McEaohern & Fowler, 1982). 

Feedback on Study Results 

We made efforts to provide feedback on the results as they 
became available. Students involved in the studies received both 
individual and group feedback on the results, in oral 
presentations and written reports (Mentkowski , 1981a, 1981b; 
Mentkowski S Fowler, 1981). Throughout the four years of her 
participation, and as an alumna in the followup studies, each 
student participating in the longitudinal studies received 
consecutive, individual feedback and interpretation of her score, 
and group results on one of the measures she completed 
(Mentkowski, 1981a). The chairperson of the Division of Nursing 
made a series of presentations to professional groups on the 
results of the study of the performance and perceptions of 
nurses, and copies of the report were distributed to the 
institutions involved. Copies were also distributed to attendees 
at dissemination sessions. A final report summary was mailed to 
each organization executive and manager participating in the 
study of management professional perceptions and performance 
(Mentko wski, O'Brien, McEachern & Fowler, 198*5). 



Rate of Participation Achieved 



We found our efforts to involve faculty, students and 
professionals and their organizations to be very successful. 
While such attention to creating procedures involved a large 
amount of staff time during the data Collection which occurred 
over a five year period from 1976 to 1981, and added to the time 
and costs of the research effort, such effort was rewarded in 
high participation rates. First, student participation rates 
ranged from 83 to 99 percent across the three separate 
longitudinal assessments over a five year period (Mentkowski X 
Strait, 198^)- Participation rates for the student perceptions 
study (Much Mentkowski, 1982) were overall, 99 percent. 

Eighty-nine percent of the alumnae contacted two years after 
college participated (Mentkowski, Much <S Giencke-Holl , 1 9B 5 ) • 
All three institutions involved in the study of nursing 
performance participated when contacted, as did 100 percent of 
the nurses invited to be interviewed ( Mentko W3ki. , "DeBack, Bishop, 
Mien ft Blanton, 1980). Fifty- three of the 55 organizations 
contacted for the management study agreed to participate as did 
94 percent of the managers and executives contacted (Mentkowski, 
O'Brien, MoEaohe rn ft Fowler, 1982). 

The rosea r-h described in this report was conducted over a 
period of five years. The f.-iet that we could continue our wo r ' 
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with students in the community of a small college and in the 

larger professional community over a long period of time and 

initiate and maintain participation is support for the 
effectiveness and ethics of our procedures. 



Choosi ng, C reatin g, Va lidat ing 
and_ Scor ing In struments 

Characteristics of Instruments 

In the section "Defining and Assessing Outcomes at Alverno 
College," we discussed the importance of .Selecting frameworks and 
measures for validating outcomes that match, as nearly as 
possible, the goals and assessment theory of the Alverno faculty, 
instruments that we chose or created for each of the several 
research objectives were derived from curricular objectives," 
principles of assessment, and characteristics of assessment 
techniques that have been identified by the faculty. For us, the 
faculty's definition of competence as developmental, holistic and 
generic, and the principles of assessment (Alverno College 
Faculty, 1979) are a cornerstone in choosing and creating 
instruments . 



Our validation instruments must reflect the general 
characteristics of the faculty's techniques for assessing student 
oerformance if we are to adequately validate student outcomes. 
T 4e must resist the temptation to import ready-made instruments 
currently available to assess outcomes and simply adopt them as 
validation tools. 



A program evaluation instrument, like an assessment 
instrument, should have the following characteristics. 

• The stimulus is valid in that it measures the 
learning objectives for a competence level or 
the broad ability being studied 

• It elicits the full nature of the ability—a 
hoi is tic process 

9 It allows an opportunity to integrate content 
at 'in appropriate level of sophistication 

• It aliows measurement of the integration of 
a competence with other relevant abilities 

• It is designed as a pro d uo t ion task rather 
than a recognition task 



» its mode is similar to the ability as usually 
expressed, rather than an artificial mode 



• It will most 1 ikely be subjectively scored, 
by more than one assessor, against objective 
criteria 



• It can be administered externally to the 
learning situation — for example, in the 
Assessmen t Center 

• It is diagnostic, because the student expects 
structured feedback as an intrinsic part of 
every experience 1 in which the college asks 
her . to demonstrate her abilities 

• It provides evidence for c reden tialing the a 
student's performance (Mentkowski, 1980) 

While performarice-ba3ed curricula ar .. likely to employ 
criterion-referenced measurement techniques, Alverno's 

student-centred curriculum also creates measures that elicit a 
range of individual differences to provide adequate information 
on the unique way each student demonstrates her abilities. Such 
information is particularly useful for diagnostic student 
feedback. Thus, instruments may also be designed to measure a 
range of student performance in meeting criteria as well as to 
provide evidence that the student wa3 or was not credentialed. 

While production type tasks usually generate qualitative 
results, we rely on generating both qualitative and quantitative 
data for responding to t the range of research questions. The 
Instruments we selected or created for measurement of each 
component of the validation model are indicated in Figure 4- 



Prod uc tio n_ J^I ensures 

Tvo forms of organization were explicitly built into the 
selection of instruments: the production vs. recognition 
characteristic and the developmental continuum characteristic. 
Both characteristics stem from Alverno assessment theory , 

The task characteristic of production versus recognition has 
been given a thorough treatment by McClelland (1980) though he 
refers to them a3 "operant" versus "respondent" measures. The 
basic issue is that, across many kinds of research questions, 
instrument tasks that in some way ask the participant to respond 
in the terms of the test developer rather than create o^ produce 
a response, have been poor predictors of future behavior of 5 the 
person. Recognition measures test the investigators' reality, 
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but not necessarily the reality of the participant, and it is 
usually the participant we want to know something about. While 
this perspective guts a higher value on operation or production 
measures, a more neutral view would still hold that the two types 
of measures assess different things, so there is more to learn by 
using both types of measures. We have intentionally used both 
prod uc tic and recogni tion measures. 

Because our criteria for instruments demand proactivity and a 
s el f-gene rated response on the part of the student, we have 
chosen the interview as the most unstructured production task. 
Interviews that measure student perceptions need to be derived 
from a student's thinking, and we have designed our own 
Interviews for use with Alverno students and alumnae. Interviews 
with professionals have followed a standard critical-incident 
technique (McClelland, 1978) that is part of Job Competence 
Assessment (Klemp, 1978), as has our instrumentation measuring 
learning to learn (Experiential Learning Log). 

Developmental Measures 

Some of our measures were designed by developmental 
psychologists to explore developmental phenomena, but others were 
designed for other purposes. Our beginning assumption is that 
performance on every measure has a developmental component, but 
that there are aspects of the task which are affected by 
non-developmental experiences and abilities. 

External Criterion Measures 



A major task in this research was to select a battery of 
external criterion measures (Human Potential Measures). Measures 
of college outcomes have come under fire as measuring knowledge 
without performance, and as unrelated to future performance after 
college (McClelland, 1973). In fact, we have not been able to 
identify any one external criterion measure that provides a 
perfect match to any of the abilities we are validating. Given 
our criteria for instrument characteristics, particularly that 
they should be production tasks in order to measure the learner 
in action, few measures meet either the demands for the holistic 
nature of the ability or the mode of measurement. We have found 
that internal validation is best carried out with faculty 
designed generic measures: generic instruments measuring Valuing 
and Communications (Friedman, Mentkowski, Earley, Loacker & Diez, 
1980), and Social Interaction (Friedman, Mentkowski, Deutsch, 
Shovar A Allen,, 1982), the Integrated Competence Seminar 
(Assessment Committee/Office of Research and Evaluation, 1982), 
and the Six Performance Characteristics Rating (Assessment 
Committee/Office oi Research and Evaluation, 198.5). 
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External validation is most effective with measures of broad 
outcomes ( cogni tive development, learning styles and generic 
abilities). As external criterion measures, we selected 
instruments that assess broad outcomes. Some 

cogni tive- developmental measures t and recently developed measures 
of generic abilities and learning styles (e.g., the Cognitive 
Competence Assessment Battery developed by McBer and Company), 
have more nearly met our criteria for instruments, and allow us 
to "talk to" researchers and theorists outside the college 
through the cor.unon language of test scores and quantitative 
results. 



One advantage of using criterion measures that have achieved 
some reputation is that other colleges are also participating to 
some extent in collecting data on students. As members s of a 
consortium of colleges, we cooperated with McBer and Company who 
have administered' nany of the instruments, used in this study to 
students at h ringe of colleges and universities with both highly 
selective and more open admission practices (Winter, McClelland # 
Stewart, 1981). James Rest ( 1979a, . 1979b) maintains a. 
clearinghouse or. Defining Issues Test data and Jane Loevinger has 
published college student norms on the Sentence Completion Test, 
which are useful in comparing changes of Alverno students with 
those at other colleges. Other norms for the Perry scheme 
(Mines, 1982) and Kohl berg's stages are also available (Kohl berg, 
1981 b). 

In addition to the instruments that employ production type 
tasks, we use a variety of ratings and questionnaires. The Six 
Performance Characteristics Rating, the Attitude Survey, the 
Management Performance Characteristics Inventory, >nd the Student 
■and Alumna Careering Questionnaires are examples of these types 
of measures. Scudont registration and assessment records provide 
a r^nge of information on student progress through the curriculum 
and performance on generic assessments. 

Create and Validate Instruments 

We had heavy invol-vement in creating and validating soir.e of 
the instruments we used. Except for those instruments created 
for the nursing and inana'gfemen t studies, most of these instruments 
are designed to provide measures of college performance, 
attitudes or perceptions. The following instruments were created 
for the specific purposes of this study: 

• Six Performance Characteristics Rating 

• Alverno College Attitude Survey 

• Alverno College Student Perspectives Interview 
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In addition to this instrument development work, we also 
•'"a"! i '.',o<1 that the .atato of the arc in developing college outcome,-? 
"i * 1 a" \ i os wan such *-'»at we could expect to either contribute to or 
work towari the validation of the instruments we were using as 
"Sternal criterion measures. First, we validated both the 
Int'\grat-»d Competence Seminar (Assessment Committee/Office of 
*-:esear;h ah' Valuation, 198;?) and the Six Performance 
■ "harao ter is tics Rating (Assessment Commit tee/Off ice of Research 
and Kval.ua t ion, 198'5). The Alverno College Attitude Survey 
{ Mentko wciki , 1977a) wan tested for reliaoility and differert 
f o rm s we re c r e;i t * d f o r students in each of t wo major p ro g ram s , 
and one type of scaling was compared against another (Mentkowski 
\ Doherty, 1979)- The Alverno College Student/Alumna 
Perspectives Interviews (Mentkowski S Much, 1980a; 1980b) were 
revised. The student interview was created after initial pilot 
work where students were interviewed with an even more 
unstructured open-ended interview (Mentkowski, 1977b). The 
alumna interview wrs revised following the first five interviews 
t o c 1 a r i f y the questions. 

In regard to the Hunan Potential Measures, we conducted an 
extensive validation of the instrument measuring the Perry scheme 
( Mentkowski , Moeser £ Strait, 1983)- We carefully described our 
procedures for establishing and maintaining the reliability and 
validity of the ratings for the Sentence Completion Test of ego 
development ( Mentkowski, Miller, "Havies, Monroe # Popovic, 1982). 
We collaborate with McBer and Company on the Cognitive 
"ompetene^ Assessment Battery by exchanging data, so that both 
Alverno are] McBer had access to the most up-to-date information 
on the validity of the measures. And we collaborated with David 
Kolb arid Olen Gish on the validation of the Adaptive Style 
f n v e n t o ry > 



We employed three general strategies for ensuring the 
validity of our instrument scores. First, we trained the 



comp rabii. L ty of the administration of the instruments, 
especially over the five years of a longitudinal study 



Score Ins t rumen ts 



a d m i n i s t 
the pro- 



of the instruments, and kept extensive records on 
es for administration 30 that there would be 
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' Men Uo ws k i ft Do he r ty , 1 977 , 1 r »7 f i , 1 980a , 1 '180b ) . Wo a 1 so 
nrovided our Interviewers with extensive training, and also 
trained our coders of qualitative interview data ( Mentko wsk i. , 
DeBack, Bishop, Mien ft Rlanton, 1980; Mentkowski , O'Brien, 
McEachern ft "owler, 198:>). Further, we consulted with instrument 
designers and tn*Lr colleagues for all but one of the instruments 
('Watson ft Planer, 1964), and participated in workshops that 
provided training for the coding of data from the Measure of 
Vocational, Educational and Personal Issues by Lee Knefelkamp, 
for the Behavioral Event Interview by. George Xlemp and iJavid 
McClelland , and fo r the Sentence Completion Test by Jane 
ho evince r . 



We knew from the oucset that using production type tasks as 
college outcomes measures would require a large effort in the 
scoring and/or coding of the instruments. In order to accomplish 
this task, we used expert scorers outside the college for 
assistance. Scorers at MoBer and Company directed by Ann Li twin 
completed scoring of the Analysis of Argument, Test of Thematic 
Analysis, Picture Story Exercise and Life History Exercise, and 
Initially for the Tert of Cognitive Development (see Winter, 
McClelland and Stewart, 1981, for details). The Moral Judgment 
Instrument was scored by John Gibbs and Clark Power from the 
Center for Moral Education at Harvari University. The Measure of 
Vocational, Educational and Personal Issues (after Perry) was 
scored at Alverno, with a workshop frcm Lee Knefelkamp and 
further input from William Perry, since Alverno was engaged In an 
extensive validation of the process for judging student 
performance on the Perry .scheme (Mentkowski, Moeser ft Strait, 
19.85). The Sentence Completion Test of ego development was also 
scored at Alverno with input from two scoring workshops conducted 
by Jane Loevinger at Washington University (Mentkowskl, Miller, 
Davies, Monroe h Popovic, 1982). 



The Learning Style Inventory and the Watson Glaser Critical 
Thinking Appraisal were scored by hand at Alverno. The Defining 
Issues Test' and the Adaptive Styie Inventory were computer scored 
at Alverno with programs provided by James Rest and David Kolh, 
respectively. Alverno sc^-ed the performance characteristic:, 
inventories for the management and nursing studies with 
consultant assistance from George Klemp and David McClelland, and 
Alverno coded data from the careering questionnaires. 



Throughout the work, we experimented with various ways to 
analyze the ;nen-ended interview data from the study of 
student/alumna perceptions. Our methods ranged from creating a 
codebook specifying developmental Levels of categories and 
examples, to a detailed analysis using all relevant parts of the 
data related to a question or category, to reading selected 
interview examples and generating a description of the overall 
findings. An outcome of this work Ls that we confirmed that an 
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indeplh Mn-ilysi. ', of the material required 
background in qualitative dnt-i aria 1 ys i s . 



a soe Lai so Lenoe 



During the -'.onr.se of .iforLn^ the ins t run* u ts , we created two 
detailed nssossm-n t proc.M:ji?;', and dM extensive work to Insure 
their validity. One was created in collaboration with Lee 
Knefolkanp for scoring essays Tor the *Vrry Scheme ( Mentko wsk i , 
Moeser S: Strait, 198'0, ^nd one process was created- in 
collaboration with George Klemp for deriving competences from the 
Behavioral Event Interview from the nursing study (Mentkowski, 
BoBnck, Bishop, Mien fk Blanton, 1980) and for coding the 
Behavioral Ev^rit Interview (MoBer S Company, 1978) for the 
management study (Mentkowski, O'Brien, McEachern a Fowler, 198;?). 



Throughout the scoring and cod ing of all the data, we 
maintained contact with the instrument designers with one 
exception f Watson and Glaser) . We recognized that we needed to 
keep up to date with the latest information in the validation of 
the instruments, but more important, that the measures themselves 
could benefit from vie results of a five year longitudinal study 
employing them. These results can greatly enhance our 
understanding of the meaning of the instruments because they were 
given as a battery and could therefore be interrelated. This is 
particularly important since many of the instruments are just 
being developed. In addition, we can provide data on women's 
abil i ties . 



Another outcome of this extensive work is that we have been 
able to disseminate some methodology useful to educators. The 
criteria and process used to judge student performance on the 
^erry scheme (Mentkowski, Moeser i Strait, 1983), the Behavioral 
Event Interview process (Mentkowski, DeBack, Bishop, Allen & 
Blanton, 1980; Mentkowski, 0'3rien, McEaehern & Fowler, 1982), 
and the strategies for interviewing students have all been 
effective in various other projects here at Alverno (Schall A 
Ouinn, Note 4), ^nd some other campuses. 



Selec t Ba ta Analysis Strategies 

Lata analysis strategies were chosen following reviews of 
available methodology. James Rest and *' T)avi3on of the 

University of Minnesota and Marcus Lieberman 1 the Center for 
Moral Education provided us with several suggestions and insights 
useful in the analysis of the longitudinal data from the Human 
Potential Measures, George Klemp guided our analysis of the data 
from the studies of professional competence in management and 
nursing. Finally, our Evaluation Advisory Council, Donald Grant 
of the University of Georgia, Mil ten Hakel of Ohio State 
University, end Joel Moses of ATftf, aided us in the validation 
and development of our college performance and performance 
characteristics measures ' Integrated Competence Seminar; Six 
ri harac terist i cs Rating") . 
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I n s t ru m ° n t T)e sc r i p t_lon s 

A brief description of each set of instruments Ls Riven 
below. Re.senrch reports describe th^ inst runon ts in more detail. 

Humeri Potential Measures: Cognitive Development 



".'est jf Cogniti ve Dev elopmen t , ( 

TRenV^r"rVt TlV f 1976 f afteV'lnhclder .5 Piaget, 1958) 

3v hnvinp; -j strident work a series of problems and provide 
reasons for answers , this insmr—nt' measures a student' s 
cognitive activity based on Placet's stages of cognitive 
development. The measure is more narrowly focused on a single 
.•stage of cognitive development, formal operations. 

Sentence Completion Test (Loevinger, 1976; Loevinger, Messier ^ 
P~ edmoV e 7 ^970 ; Lo ev in g er 3 Wessler, 1970) 

A production task elicits a measure of an individual's stage 
of ego development. Ego here is' defined as one's style of life, 
the unity of personality, individuality, the method of facing 
problems, opinion about one's self and the problems of life, and 
the whole attitude toward making choices in all life spheres 
(Loevinger $ Knoll, 1983). 

Moral J udgmen t Instrument (Kohlberg, et al., 197M; 
Kohl berg, igsfa^ 1981b; Colby, et al., in press) 

This production task elicits response to a moral dilemma. 
The instrument provides a measure of an individual's stage of 
moral development by analysing the re zoning a person gives in 
response to questions that probe reasoni', about moral issues and 
orientations that create and define moral dilemmas. 

De fining Issu es Test (Rest, 1979b, 

best's instrument (based on Kohlherg's theory of moral 
development) provides a measure of an individual's moral 
development in a recognition task by analysing the relative 
importance attributed by a person to principled moral 
considerations. k person attribu L es importance to several 
reasons given for resolving a particular moral dilemma, and then 
rank orders them. 



86 76 



Mea.jur'> of Vocat iona l, Edjic -\tJjDnnl ,_and Personal Issues 
T^nefelkampV Vr 7 4 ; Wiii r k, ~1?7 r f; "^vi3^^for ^ l^^f^<a^ p < 
Stepit:;u, 1 ^^6 ; after Perry, 1970; nov> titled the 
Meas ure of Intellectual ^Deve lopmen t; Minos, 1 )82) 

This production task measure of the Perry scheme of 
intellectual and ethical development asks students to write three 
essays describing "the best class you've taken...," "a decision 
about something that had major importance...," and "things oa 
cons L i e r wh c n a pp ro ac hi ng th e q u cs t lo n of career choice... . " T v ^ 
essays are judged for posi tion on the Perry scheme <; 
intellectual and ethical development, using the Alverno Criteria 
(Meritkowski. , Mo^aor ■$ Strait, IQfti). 

Human Potential Measures: Learning Styles 

we a r-n i.ng_ Style Inventory ( K o 1 b , 1976) 
Adaptive Sty l~e J nv_e_ ntory ( K o 1 b , 197*3) 

7 he Learning Style invento ry is a measure of individual 
learning styles which affect decision-making and problem-solving. 
The four styles are Concrete [experience, Reflective Observation, 
Abstract Conceptualization, and Active Experimentation. This 
recognition task requires trie student to rank order descriptive 
statements about her .mode of learning. The Adaptive Style 
Inventory measures the extent to which the person is likely to 
use each mode in various situations, and assesses fo/ adaptive 
competence through a recognition task. 

Life His tory E xercise (Klemp * Connelly, 1977) 

This instrument using a recognition task is a measure of 
interpersonal learning skills. The cases are programmed in such 
a way that a person with good judgment about people (i.e., one 
who does not make snap, impulsive judgments) will become more 
accurate in choices of the correct alternative as the respondent 
proceeds through the case. The instrument assesses how one uses 
Information in making decisions about others or predicting 
behavior ani examines the process by which decisions are made. 

Hu ma n °otential Meas u re s : Ce ae r i c Ab i L i t i es 

Anal y s ls of _j A r gu m e n t (Stewart, 1977a) 

This instrument is intended to assess intellectual 

flexibility by requesting the student to argue against a 

controversial opinion, and then defend the opinion just attacked. 
The measure uses a production task. 
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Test )L' Thematic Analya^Lo 

T¥Tot ^"r ; Tf7i~ W inter ~% McClelland, M7M) 

This instrument consists of two sets of three stories, 
students are asked to compare the sets tnemat.ieally, * .production 
task. This "thematic analysis' is scored according \o twelve 
categories of critical thinking- This Ust is ba^ed on an 
understanding oi cognitive development defined as the ability to 
analyze new information and to synthesize new concepts based on 
this' information, and reflects the ability to integrate 
information into one's own cognitive structure. 4s the cognitive 
structure grows, so does the ability to think critically, to make 
a cogent argument 'and to reason inductively. 

Picture Story Exerc_; Winter, McClelland ft Stewart, 1981 ) 



The instrument requires the student to write narra es to 
six pictures. This instrument, modeled on the Tuematio 
Apperception Test (Morgan S Murray, 1935), is used tc assess i 
variety of abilities. One is "self-definition" which emcompasses 
th3 vray one thinks about the world and one s self, the way one 
reacts to new information, and the way one behaves (Stewart <x 
Winter, 1974). People with high cognitive initiative are not 
only able to think clearly, but also to reason from problem to 
solution, and to propose to take effective action on their own. 
This instrument is also used to assess Need for Achievement 
[ McClelland, et al., Affiliation (Atkinson, 1958), Power 

(Winter, 1973) and Activity Inhibition (McClelland, 1975). 
Stages of Adaptation, a measure of ego development created by 
r.t'-wart (l°77b, 1982) after Freud Erickson, are also scored from 
the Picture 3tcry Exercise. 

'rfatson-Glaser CrUi^al^hi^ < P < Glaser, 1964) 

This is 'i traditional and time tested recognition task 
measuring several components of critical thinking. Inference, 
Recognition of Assumptions, and Deduction were used in the 
current study. 



Me—.ures of Student Performance 
in the Learning Process 



Progress (Me "i * Strait, 193*5) 

Several L: 1 ir itora of progress through the curriculum are: 
- e- of semesters attended; number of credits achieved; and 
« r of competence level units achieved at any one point in 
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ERIC 



Integ r ated { ' on o»j t e n ne Seminar 

( Assessment ^o.amTu^^0TncV of Research and Evaluation, 198?) 

This assessment technique provides an opportunity For a 
student from any discipline ^ demonstrate integration of her 
abilities develop.'! by the mr'i>...- .. of her college education. 
The student shows how she trv r ^:rs these abilities to a new and 
complex simulated situation she is likely to face as a 
college-educated v.*. a. The indent is not credentialed on this 
instrument; It ,u e l for diagnostic purposes only. Thus, the 

1CM is, in part, an external criterion measure of the student's 
ability to transfer her learning after completing the general 
education sequent-. The student performs three exercises over a 
four hour perio j as a member of a decision-making board of 
o i t i zens : Oral " osen ta t ion , Tn-Basket , and Group Discussion. 
Off-campus professionals serve as assessors, who individually 
observe and evaluate each student's performance against specified 
criteria, come to consensus, prepare a written evaluative 
statement, and meet individually with each student to provide 
feedback on her pe r formanee . This instrument was revised in 
1981 . - 



Six Per formance SK^^J^Sl. 3 } K c s Hat: ing (Assessment Committee/ 
Office of Research and Evaluation, 1978, revised 1979, 198?) 

Six cross-disciplinary performance characteristics were 
identified and defined by the faculty (Alverno College Faculty, 
1977) to describe the student's developing ability to interrelate 
and internalize performing the competences. At the present time 
the faculty have defined five characteristics which apply to her 
behavior-- Integration, Independence, Creativity, Awareness, 
Commitment— and a six th — Habitual ity— -which modifies the others. 
The development of these characteristics takes place initially as 
the student strives to acquire or to improve abilities demanded 
Ly her discipline or profession. Gradually, the characteristics 
themselves become central to her style of working and to her 
exercise of personal responsibility. Faculty think of these 
characteristics as contributing to her personal and professional 
life, and incorporate assessment of them into a longitudinal 
evaluation program. F>ach first and second year student in the 
Weekday College is rated on each of the characteristics by an 
instructor at the end of each year in college. Each third and 
fourth year student receives two independent ratings from two 
professors in her area of specialization and one rating from a 
profess:! 1 in her minor area. 
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Measures of Student/Alumna Perceptions 
of Learning and Career ing 



■\ 1 v e r no Ooi lege S tuden t Per a pe - tiv es_In; ter v ie w 
i^entkoiraki" ft Much," 1980b; fn Much ft Mentkowski, 1982) 

This interview is designed to enable student.^ to speak for 
themselves about their college experiences. It measures 
students' perspectives on many aspects of college learning. Tt 
is open-ended and probes students' thinking, asking them to 
describe their perceptions of learning, how they have changed, 
.nd'whv. Questions that focused the design of the interview 
questions are also included. 



Uverno Coll_e^e_AJ^mnn ^ej^£^tiv_e3 Jj^JiLview 
TSentiroirakT* Much, 1980a; In Mentkowski, Much 



enc ke 



-Moll, 198.5) 



Similar in 



form to the interview for students, this interview 
r ocu3 ,, on an alumna's perspectives on transfer processes, new 
learning, careering and professional development and integration 
0 r personal and professional roles. The open-ended questions and 
probes ask her to draw relationships between college and her life 
a a it is now. Design questions are included- 



\\ ve rno C o liege At t i t ud_e j^r v_ey 
FoTnTT~and Form bTTMentkowski , 1977a) 



The Attitude Survey is an objective questionnaire measuring 

students' perceptions of and attitudes toward a variety of areas 

3UCh as: "Alverno," "Faculty," "Changes," Syllabi, ^ 

"Performance," "Professional Development," "Assessment Center, ( 
"Competence Level Units," "Challenge," "Progress," "Social Life, 

"AdvisiT " "Ca-eer Development," "Need for Structure, Learning 

Modes," ' "Confidence," "Library, Student Activity, Residence 
Hall," and "General Issues." 



Alverno Co liege Stu d ent Career i n g J^jrtj^l*\fJi 
Xfet^owski * Bisho'p, 1 qR 1; In Mentkowski, Much ft 
Giencke-Holl, 1983) 

This careering questionnaire is administered to graduating 
seniors and includes questions in an objective format that 
requests information on the following: paid and unpaid wor< 
experience before, during and after college; field experiences, 
clinical experiences, student teaching and participation in 
Alverno' s Off-Campus Experiential Learning Program; salary before 
college and now; expectations for getting a job and changing 
iobs- expected job area and title; expected salary range; ratings 
of -oil eg- preparation for futur job performance; expectations 
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° r IV.:!!-.' j->h satisfactions and potential for advancement; 
»?xpeo tat ions for performance on job- related exams; expectations 
for continued schooling; self-description of motivation to learn; 
rating of the effectiveness of the Alverno experience; 
3at isfaetinn with current choice of major or career; expectation 
of advantage of a college r!.^n»o in Retting a job; ratings of ->7 
general and specific goals expected to result from college on 
thr^p variables (goal importance, goal preparation from Alverno, 
and current goal achievement); parents' occupations, number of 
?hildren, marital status. Attitudes toward working and sources 
for financing college are also included in the revised 
i nstrunent . 



Mv^rno College Alumna Careering j}u_es tio nna i re 
TMentkowoki 4 Bishop, 1980a; "in Mp7nFko v^Ti^H^uch - e < 
Giencke-Holl , 198*3) 

This careering questionnaire includes the questions contained 
in the instrument for Alverno graduates, but it is worded to 
collect data on Information such as work experience since 
graduation; success in obtaining a po3iMon related to a chosen 
career; current salary, saiary increase; ratings of career 
preparation; potential for satisfaction and potential for 
advancement La the first position held after graduation; reasons 
for not seeking paid employment after graduation; performance on 
exams related to obtaining . job; continued education; 
satisfaction with college career choice; ratings of goals no*; 
membership in career- related associations; and membership in 
civic or other organizations. 



Measures o f Prof ess ional Performance, Perceptions, 
and Careering and Professional Development 

Be havi oral Event Interview 
IK! em p, Tq^sTlfcCl eU and*" 1 976 , 1 978 ) 

This critical incident technique eli\y£s six situations in 
which the professional identifies as effective or ineffective. 
The professional, guided by the interviewer, describes each 
situation, who was involved, and the outcome, but primarily 
focuses on what s / he actually did in the situation, and what 'n/he 
was actually thinking, feeling and intending at the time. 
Interview transcripts are used to create the Behavioral Event 
Interview WrLteup from which competences can be derived or coded. 
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Be ha v i o ra 1 Ev e n t Interview W - i t eup 

XTfv erne" ^oTle^T^O fYi'oe oT Ite s e aY :h a n d Ev a 1 u a t i o n , 1 9 BO ; a f t * ? r 
McClelland, M" 7 ^; In Mentkowski, O'Brien, McEachern ?i 
Fowler, 199:0 

Interview content from the Behavioral Event Interview 
transcript is summarized in written form as soon as possible 
after the interview. The form calls for a description of each 
incident, the components of the incident, the participant's jot; 
responsibilities, and the characteristics 3/he thinks necessary 
fo r out stand in ft pe rformance. Fo r each inc ident , the 
participant's behavior is described in detail, and what she was 
thinking, feeling and intending at the time. The result or 
outcome is also described. Information that would identify the 
participant is deleted. 

Nursing Norn i na t i o n Que s t i o n na ire (after Klemp, 1978) 

The nomination questionnaire is a two-pafte instrument that 
briefly describes the study and asks participants to list those 
professional peers whom they consider to be "outstanding." Space 
is allotted for ten names- Participants are asked to list as 
many "outstanding" peers as they can from memory. 

Nursing Job Elemen t Inven tory (Grau b Rutter, 1977) 

The Job Element Inventory is comprised of a list of 120 
performance characteristics nurses identified as necessary for 
" -utstanding" or "superior" job performance. The purpose of the 
inventory is to ascertain the behaviors/ characteristics 
participants think nurses must possess for outstanding nursing 
performance. Participants respond to the list three separate 
times. They check those *t>ehav io rs they believe (O distinguish 
"outstanding" from "good" nurses who share the . r job title, (2) 
characterize "marginal" nurses who share their job title, (3) are 
more important in hiring or training for their job. 

Nuralnft C areeri ng Que 3 tin nna a n e (Grau, 1977) 

In objective format, the questionnaire elicits information 
about marital' status, number of dependents, year of licensure, 
years of nursing experience, type of educat onal preparation for 
licensure, current educational pursuits, an estimate of future 
educational endeavor job satisfaction, and self-evaluation of 
job performance. ■ measure was used to categorize interview 

participants on educational background , years of nursing 
experience, hour^s of employment per week, marital status, job 
satisfaction and self-perception of- performance. 
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r, *an ! igern*!,nt :) o rfo nuance ''har'U': teristi.es Inventory 

IB i 3 h cyp~V e n tkows YiT "0 fen" ,~ XiTnoyT T)av fes'T "MV. Ran ho rn, 1 980 ; 

In Mentkowski, O'Brien, ^IcEaehern A Fowler, 1982) 

Rach manager is asked to judge on oh of 162 statements or 
performance characteristics (1) as relevant to one's own work 
experience, (?) as »■ • tibial to' selection and training, and (3) 
as characteristic of outstanding performers. Characteristics 
that meet 'ill three criteria for judgment are then considered to 
be descriptive of effective management performance from the point 
of view of the managers studied. Through an item scoring 
procedure, characteristics that discriminate average from 
ou t s t an d 1 ng pe r f o r m or s a re i 1 en t i f t ed . 

Management Careering Questionnaire (Mentkowski p < Pishop, 1°60b; 
Tn ^en£ ^o'wsYTT ~0 ' 3 r i er^l'IcEa c he rn A Fowler, 1982) 

This measure collects information on several variables 
r e 1 a t e d to 0 a r e e r L ng an d p r o f e s s ; o na 1 d c v el r. , at. I n f o r m a t i o n 
is gathered from the raana/-,^' on job responsibilities and 
function, who the manager reports to and who reports to the 
manager, position level and type, experience in the company and 
the last position the manager held in her previous company. The 
manager also reports levels of education completed and in 
progress, area of specialization, and completion of a management 
training program. Number and breadth of professional activities 
not soon so red by the company are also indicated. Personal 
information includes marital status, number of children, number 
of roles, husband's occupation, mother's and father's occupation, 
parents' occupational status, number of siblings, and birth 
order. 
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WHAT ARE THE OUTCOMES OF A COLLEGE EXPERIENCE? 



Because this is an cv. going project, consisting of 
interrelate studies, wo do not anticipate one sot of "final" 
results. Most of our studies are continuing and vri.ll continue 
for 30ver.nl year;. What wo can report, after seven years, are 
preliminary results in our major area of inquiry. For this 
paper, v, discuss results from the ten research reports under two 
major healings, "What are the Outcomes of a College Experience?" 
and "How Do College Outcomes Relate to the World of Work?" 

We have also been able, ' al ong trie way, to make 130m e 
contributions to the newly developing field of educational 
nr-.-Tiin evaluation and to the repertoire of procedures for 
val .dating developmental outcomes. Tt is for this reason that 
tills report has included an overview and summary of the research 
methodology as well as the rationale and results. In addition, 
our work seems to be offering some substantive support for the 
.-. f oe]s of out come- cento red curriculum design. 

We have taken three independent approaches to measuring the 
meaning and development of the broad abilities college is 
expected to foster. These are: (a) student performance on 
college-designed ability measures within the curriculum, (b) 
student performance on a battery of measures from outside the 
college describing human growth patterns in cognitive 
development, learning styles, and generic abilities, and (c) 
student perceptions of the reasons for learning, the process of 
learning, and its value for their cwn career and Life goals, and 
the perceptions of alumnae two years out of college. Results are 
based on longitudinal and cross-sectional studies with over 750 
won e a s 1 1 id e n t; 3 a g ed » 5 • A core g ro u p of over 2 00 of the so 
make up the longitudinal participant group. 



We have seen significant change, in all three Kinds of 
measures. These changes confirm that students do change in their 
developing cognitive patterns and abilities — whether these are 
defined by faculty or by other pr- titioners and researchers 
and that their own ' perceptions and uses of learning change 
concurrently. 

Student Change on College Defined Abilities 



Performance in the Learning Process 

Students have consistently shown Change on the college's own 
measures designed by the faculty. Each graduate has, along the 
way,- engaged in more than 100 active performance assessments in 
her various courses. Faculty design each assessment to elicit a 
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particular level f one of eight major abilities, using the 
course's discipline- content as a context. Each graduate's 
performance:.? have been variously assessed by faculty, peers, and 
community professionals (and always by herself), according to 
criteria that remain stable across all disciplines. 

We think it is important that so many students have shown 
consistent change through this complex network of performance 
measures. It suggests that the complex outcomes, identified by 
the faculty are indeed developable, and vis ible in per formance to 
both faculty, student-, and profession.**. -S from outside the 
college; that a compI-.-x ability is recOi g rr ifla bl ^e across settings, 
despite the varied forms it may take in different disciplines and 
professional environments; and that such' Abilities can be 
developed sequent i ally to increasingly complex levels. 



/ /Ire Measures Valid? 

/ Is Expert Judgment Reliable? 

--From a v aliSatioa^ point of view, two questions- immediately 
arise. \re the college's assessment measures themselves valid? 
And do the faculty indecc share the kind of consensual perception 
of 'student performances, the inter-rater reliability, so that we 
can be satisfied that the progress students make^ is actually 
there? 



Inte grated Ab i i/i't i e s 

Before 9ur validation project began, the faculty hfed already 
moved to design certain out-of r cl*iss assessments .thaj; would 
function as external criterion measures. At the^mid point of her 
college career, for example, ttrey recuired each ; student- to 

'.participate in a half-day interactive simulation called the 
Integrated Comptence Seminar (ICS), designed to elicit her 
performance of five major abilities at once. The ICS rated by 

.expert, judges , is a "content- fair" crois- disciplinary measure .of 
the' abilities students were developing and demonstrating inHheir 
several courses (Assessment Committee/ Office Res'each and 

Evaluation, 1982).| Assessor judgments were reliable, 'but the 
three exercises ! that make* up' the simulation differed in 
difficulty arid validity. The "In-Basket, a measure of Analysis 
and Prob^e^-Solving worked best. The measure of Social 
Interaction, judgefl frorrf video taped group discussion, did not 
correlate in the expected directions with other measures. Social 

'interaction is, ap ability that has only recently been developed 
through systematic^ instruction; perhaps we , can ' t -get a handle on 
an adequate measure yet'- The importance^!' this ability for 
future performance' at work was underscored both in studies of 
alumnae nnj p^o/^ssionals. Partly as a result of this st»^dy, 

* faculty are currently testing out a substantially revised /CS, 
and including demonstration of all eight major abilities. 
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"vacuity Rating, of 
Stud eat Development 



Another such measure is a summary rating faculty gi' r e each 
stud*' .t at the end of; each year in college, reflecting their 
judgment of hor overall per formance on her assessments tha^ year. 
That rating includes six fairly intangible performance 
o ha r.ac tf ■ r i 3 1 i o 2? ( A wa re n ess, Creativity, Independence, 

Integration, Commitment, -<nd HaM luality) which function in 
relation to the more tangible assessed alilities much like 
adverbs function in rel.it i.^n i ) a verb. Our five-year study of 
this - rating 'ir-, in confirms the/nigh agreement among faculty as 
-xpert judges, and indicates that it validly measures student 
change in both cross-sectional and longitudinal comparisons 
/ Assessment Committee/Office of Research and Evaluation, 1983). 
Although we have not yet been able to validate its power to 
discriminate nnon^ the six: performance characteristics, the 
rating is more highly con- 1 n ted with student performance on 
cognitive-developmental measures, than with those tha>, measure 
more specific abilities. This suggests t t faculty ratings on 
the six performance characteristics are tapping the underlying 
patterns of student development. Faculty as ^i whole recognize 
these student differences in developmental level,, and we infer 
that this awareness is reflected in instruction and assessment. 




Faculty continue to use the^rating extensively, not to 
accredit students bu*- to learn more a^ou', the characteristics, 
and to challenge their theory and practice of assessment through 
this shared assessment experience. r Yivrently, faculty are 
reviewing both the definitions of the^cha rac terist Ies and the 
judging process for 4:vs i grang ratings in pre para t ion for 

f]v- nd ipg . its use for 
trate their abil it ies. 



'in 



/ 



developing a better instrument 
describing; f ^a, unique w a y s student s d e men 

n j e n e r i o Kb Hit ies : . C or n mun i c at _i on_ s , 
Val ui ng , So c ia_l lj^t_ f ^_ r J i ^^tj °J1 

Faculty have also* created generic out-of-elass measures for. 
several of the eight major abilities. F>ach of these elicits the 
Student's performance in thai; ability at the level reouired o* 
all students for graduation (further "advanced" levels are 
required of each'student in selected ability i, depending on her 
major). So far we have studied three sucn generic measures, 
those .lor-i&ned ' for. Communications, Valuing, and Social 
Interaction- ^ 



Our study of the Communications generic instrument indicates 
that i t val idly d iser iminates instructed from uni^strueted 
performance as doe>s the Valuing generic instrument ( ? riedman, 
Ment>oW3ki', Farley, Loacker k Dies, 1980). More iraporlan.;, 
patterns of ctudent performance validate the sequen t;ial 1 ev :1 s of 

\ 

\ 
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Communications. v or the Social Interaction generic instrument, 
we again have had more difficulty demonstrating Liu 1 results that 
instructed students perform at higher levels than uninstrueted 
students (Friedman, Mentkowski, Deutsch, Shovar $ Mien, 



Creating Strategies for 
E v a 1 u a t i ng and R e v i a ing 
Inst ruments 

These studies did lead us to try out a variety of strategies 
for validating these non- traditional assessment techniques. Some 
worked better than others, depending in part on how well the 
ability we are measuring is understood- Abilities like oocial 
Interaction are new to higher education instruction, and we have 
a long way to go to adequately validate these kinds of measures. 
We have found that our older college population helps in this 
regard because we get a better picture of just what aspects 
develop through informal learning, And some quantitative 
strategies work better than others In showing differences between 
instructed and un ins true ted students. In another series of 
studies with a range of sixteen other measures (Assessment 
Committee/Office of Research and Evaluation, 1980), we found that 
criteria evaluation, establishing inter-rater reliability of 
assessors and pre- and post-instruction comparisons were three 
strategies that functioned well and were accepted as workable by 
a range of faculty from different disciolines. "Direct 
involvement of faculty in analyzing student performance data and 
probing validity questions generates a broad scope of validity 
issues. 



In sum, the faculty demonstrate high reliability as expert 
judges of student performance. We will continue to study the 
generic instruments, as well as the Integrated Competence Seminar 
and the Six Performance Characteristics Hating, since they seem 
to function effectively as the colleges dwn external criterion 
measures. This i3 more feasible than attempting validation on 
each of the hundreds of in-class Instruments, which arc 
frequently revised and which are oft„en short-lived, and may offer 
intervals for student change no longer than a single semester. 
If the faculty's own milestone measures are reliable, they can 
use them confidently in the ongoing creation and redesign of 
their in-class assessments. But do such changes also show up on 
measures designed by theorists who describe human potential in 
broad growth pat terns? 



Our major finding indicates that students have consistently 
3hown change on a battery of instruments, drawn from sources 



Student Change jon JBroad_, 
Pevelopme n t a 1 _Frame wo rks 
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o u t s i 1 e t * 1 ^ college, whi o h , a r e d o; 3 i gn o d to in ea sure v a r i o us 
cognitive-developmental pa tterns , learning styles, or 0 ther 
generic abilities (Mentkowski strait, 193'i). Twelve different 
instruments were drawn from three major frameworks. One is 
cognitive-developmental theory (Tnhelder e < Piaget, 19^; 
Kohlberg, 1981a, 1981b; Loevinger, 1976; ?erry, 1970, 1981; Rest, 
1979b). Another is experiential learning theory (Kolb, in 
press), and the third source is from a recent thrust to identify 
r -*nd measure generic abilities that link education to performance 
after college (Winter, McClelland 5 Stewart, 1981). This battery 
of human potential measures was administered to two complete 
entering classes and one graduating class (altogether about 750 
.students). The entering classes completed the same battery two 
years after entrance, and again two years later near graduation. 
Thus, we have a set of longitudinal results which can be 
d ou b 1 0 - 2 h e e k e d a g a i. a s t r e s u Its f ro m a c r 0 ss - s e c t i o n a 1 study of 60 
graduating seniors compared with entering students who later 
gra dilated. 

The data on ?00 of the students who completed the battery on 
three occasions provides a parallel stream of longitudinal 
information alongside these same students 1 progressive 
performances on college assessments. Data on all the students, 
both within and outside the longitudinal samples, also provides a 
source for various cross-sectional comparisons. 

The design includes two age cohorts to control for the 
effects of maturation, and two class cohorts to enhance 
representativeness. The time series design holds time constant 
and allows performance in college to vary, so we can attribute 
change to performance in college in the absence of a control 
group. 'We also control for age, background and program 
characteristics, when we study the effects of perfor mce in the 
learning process. 

It has not proved possible to locate measures developed 
outside the college which are readily congruent with all or even 
most of the complex major abilities that educators identify" as 
outcomes of college. The preponderance of available measures 
focus in the cognitive area, from broad developmental measures to 
instruments aimed at particular analytic thought processes. V/e 
have not been able to approach our design ideal of 'several 
external measures overlapping on faculty-designed abilities or 
outcomes. 

Most existing instruments also tend to be recognition rather 
than production oriented, so that we have had to make special 
efforts to be able to use performance oriented instruments in 
fairly early stages of their development ' Mentko wak i , Moeser $ 
Strait, 198'5). While recognition instruments are generally more 




responsive change Lndi^nMrs. 1 ■ more oconomioal to handle, our 
focus on the complex -e .lies and outcomes ->gea 

intend to foster h:. Lven us a special com. . to 

collaborating with c leagues in the emerging >i.eld of 
pe r f o nn a nc e m ea s \ x re m en t . 

Students clearly show significant development changes across 
all three occasions when the battery of twelve measures has been 
administered (Mentkowski * Strait, 198*5). Looking at the results 
of all the external instruments together, we find first, that 
students appear to change more on these external measures in the 
first two years than in the second two years. But the changes in 
the second interval are more directly attributable to the 
student's successful participation in the college's curriculum. 
This is the case even when we account for change due to the 
pretest scores, parent's education and occupation, high school 
grade point average, prior college experience, living at home or 
on campus, marital status, full or part time attendance, or type 
of major. 



Among', these other variab ige of the student may be 

particularly significant f itors attempting to ser, :;he 

"new" student effectively. On- :.- . leworthy finding here i mt 
age doe3 indeed seem to confer some initial advantages as 
reflected in the cognitive-developmental ^.scores of entering 
students, but not on the more specifically focused abilities. 
This suggests that educators can rely on age as an indicator of 
advanced ability wfth respect to broad cognitive patterns but not 
•it the more specific skill level. And age is an indicator of 
difference in learning style preference as well. 



In looking for interrelationships among the 

cognitive-developmental patterns, learning styles and generic 
abilities we measured, we have found an unanticipated but 
valuable result. When students entered college, and again two 
years later, student's performances on the battery of twelve 
measures tended to statistically cluster around two separate 
developmental factors — one we call logical or analytic thought, 
and the other we call socio-emotional maturity. But after four 
years in college, the two clusters had merged. This may reflect 
one of the most desired outcomes of college, namely, that 
students are integrating their own understanding and use of these 
two kinds of abilities. 

We expect that measures that call for recognizing concepts 
require a less sophisticated level of understanding than measures 
demanding that a student produce the concept. Recognition 
measures (ranking statements, multiple choice) should show more 
change; production measures (essay, set of stories) should show 
less. Indeed, the recognition oriented measures in general show 
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more change across four years than do the production measurer,, 
but in the second two-year interval, the trends of that pattern 
•^an to reverse.. This supports a -ecent trend (McClelland, 
, n ^^ r ,<i Stewart, 19 ; - ' ^ to d- ' ..asures of college outcomes 
./hieh ask students to gene .' .ties rather than to recognize 

or comprehend knowledge. Aft . ., these kind of production 
measures are used by faculty to assess advanced level work in a 
student's major. The cognitive/developmental measures and the 
one motivational measure also gave more indications of student 
change than did those focused more specifically on particular 
abilities or processes. Our expectation is that the 
comparatively smaller indications of change on production 
measures will loom larger in relation to v,... : -term effects 
concerning careering or future learning. 



Exam pi e s of Pat terns _in Stude nt Change 

1 These broad outcomes come to life as we examine the multiple 
patterns of student change that emerge from our look at students' 
developing abilities. Examples from the cognitive-developmental 
human growth measures and learning style changes illustra a these 
oat terns. 



' Cognitive Development 

Perhaps- the cognitive-developmental model most directly 
descriptive of college students and of primary interest here is 
Perry's scheme of intellectual and ethical development d-awn from 
Interview studies of Harvard undergraduates. This schemes- 
describes positions or phases through which students move as they 
respond to the diversity and ambiguity encountered in college 
learning. 

Our intensive study of more than 3000 essays from over 750 
students has enabled' us to develop a valid method and sets of 
.-riteria for using expert judgment to code essays ( NIentko wski , 
Moeser « : Strait, 1983), in collaboration with the instrument s 
designers. Applying the method and criteria, we found that the 
measure shows definite student change in both cross-sectional and 
longitudinal studies. The patterns of this change, however, are 
the intriguing result. 

When does this change occur? 'When change occurs is different 
depending on the area of development. Students show change in 
understanding learning roles and processes during the last two 
y-ars but students use more sophisticated modes of 
decision-making during the first two years of college. During 
the second two years, students show a decrease in level ol 
sophistication in decision-making, probably because they are 
assessed near graduation. They are making decisions in areas 
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rel -it->ri to future issues, md begin by using less complex modes 
of thinking. 



We studied three area? of development: classroom learning, 
dec isicn-making, *nd career. Students wrote c.n essay in each of 
these areas and it was rated foi Perry's scheme using our tested 
method -and criteria. Each of the three essays shows definite 
change patterns. But the change is not always straightforward, 
nor do the three patterns run neatly in parallel. This is clear 
when we see how olier and younger students compare on each of the 
three areas, and in classroom learning in particular. Two 
findings are of particular interest here. First, rate of 
devel -pment Is related to age for decision-making, and career 
understanding, but not for student's understanding of classroom 
learnlris processes and roles. Second, when development occurs 
during college, depends on the area. 

Understanding of classroom learning processes and roles is 
not related to a.^e at entrance to college. Older students are 
starting at the same place as younger students when they enter. 
But after two years, older students have changed more than 
younger students. Even ohough older students enter college with 
the same level of understanding classroom learning processes as 
younger 'students, they make more, immediate progress in 
understanding such sophisticated concepts as learning through 
multiple ways, learning from peers, and becoming independent in 
their own learning* But traditional-aged students do "catch up" 
during the last two years. 

Formal learning experiences, however, are necessary for 
enhanced performance of understanding of classroom learning 
processes and roles. Students with prior college experience do 
show more sophisticated thinking in this area when they entered 
our college. But just being older is not enough. Development 
seems tied to particular kinds of experience for all areas. 

In what areas of development do older students have an 
advantage when they come to college? In both decision-making and 
career areas, older students perform at more sophisticated levels 
at entrance to college than do younger students. Older students 
make a larger leap in decision-making during the first two years 
of college than do the younger students. There is also evidence 
that older students^ are more sophisticated in these areas because 
of specific kin;; >, ' life experiences (e.g., divorce and 
widowhood) that have prompted growth. 



ks stated earlier, all students change during the first two 
years of college in that they use more sophisticated modes of 
decisicn-making. During the second two year3 students show a 
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decrease in level of sophistication in decision-maki ng, probably 
because they are ranking decision about new areas of their lives. 
This pattern does seem, 'to suggest the kind of complex 
developmental movement noticed by ?iar-- n learner may 

revert to employing an earlier cogni ' -'hen coping 

with new challenge?, entering a new ; u. rowih, or focusing 

on a different ability. It will take considerable further study 
before we can :5av that these results document this phenomenon of 
decalage. But the possibility that we might validly record auch 
"developmental complexity is a promising one, particularly because 
our criteria and method enable us to measure the evolution of 
change, as well as stability in cognitive level or position 
(Mentkow3ki, Moeser 4 Strait, 1985). 



How does high school grade point average, a commonly used 
predictor for success in college, relate to these patterns of 
learning? Student change on any of the three ar^a3 of 
development is not related "to high school grade average when 
students enter college, nor does high school average account for 
change during college. Apparently we are describing different 
aspects of intellectual development than are measured by more 
traditional indicators of success in college. After all, grade 
point average in high school is a commonly used predictor for 
grade point "average in college. But learning to become a 
learner,, as measured by the essay on classroom learning, rated 
relative to the Perry scheme, appears to be tapiping quite 
different, more basic structures of thinking that, over long 
periods of time, do change as the result of college learning 
experiences. Perhaps it is these new understandings that account 
for students learning to learn, and we need to concentrate on 
**' f s to develop lifelong learners. 

Change also occurred on other c . ■ 1 r <3l 

measures. Principled moral reasoning, baseu on kohl u^rg s r.oral 
development theory and measured' here by Rest's Defining Issues 
Test, also showed clear evidence of student change. On 
Loevinger's levels of ego d evelopmen t , students entered college 
in transition between the Conformist and Conscientious levels. 
Students graduated at the Conscientious level or at ^ the 
transition, to" the Autonomous level. Students also made gains in 
the extent to which they demonstrated Piaget's conception of the 
logical reasoning and analytical thinking structures 
characteristic of adults. 



Learning Styles r 

Another example of the student's growing' awareness of 
learning processes is the dramatic evidence of changes appearing 
in students 1 orientations to learning styles. kt ^entrance, 
students showed marked preference for "concrete" over "abstract 
thinking, and for "reflective observing" as against "active 
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experimenting." In the first two years, they moved rapidly 
toward a more balanced pattern. By the second testing, they had 
come to rely equally on concrei ? an 1 abstract in ics, and to show 
a similar flexibil: y in choosing -'ither reflective .or active 
approaches . 

Student:* are, after all, expected to become more versatile 
and habitual abstract thinkers, and they should also be actively 
involved in their world as well as reflect on it. Analyzing the 
results further, we find that this balance among learning modes, 
which appears so dramatically ,in two years 1 time, maintains 
itself after entry into a concentrated, career oriented major. 

i 

Stud ent Perceptions of Learning 

But do students 3ee themselves as making these changes? The 
major result from this source of data so far is that students do 
show consistent change during college in their perceptions and 
descriptions of learning. . Of almost equal importance is the 
finding that 3tudents maintain a pattern across all four years of 
justifying learning in terms of its relevance to their , career 
expectations. "What changes is how they see the 1 nature and role 
of learning within thi3 stable framework (Mil oh .5 Mentkowski, 
1982). 

We gathered the student. perception data through an #pen-ended, 
interview format ranging up £o two hours in length, guided by a 
protocol of questions and probe3. -The interview itself, the 
protocols, and a method for analysis were developed as part of 
this project (Mentkowski 3 Much, 1980b). Because this measure i;s 
lengthy and complex, both to administer and to ana 1 : ',e, wn 
selected samples for inter' : e wing from b^ 1 u the long! 
cross-sec onal study p :,ions. Th«j ,ults her reflect 

about 320 interviews from 60 students who provided i: rviews at 
one year interval s throughout college; data from trad itional age 
students has been analyzed so far. These students also completed 
the external measures three times during this period, as did 
another 57 who were interviewed as seniors and as two year 
alumnae . 

From the outlet, students view learning in terms of their 
career goal -J. Trad it iohal-aged students are critical of many 
kinds of learning on." this score at first. As^they progress, 
however, students consistently- develop the ability to assimilate 
widely varying courses into their , rationales. 

By the second interview, for example, they argue that courses 
in divergent areas and abilities like aesthetic response "help me 
to stay open-minded" or "give me a broader view of things £nd 
people." Such descriptions are in "tu™ justified by 
explanations: "I know that as a manager L'll have to deal with 
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poo pi- ir.y'i 'ill ki'i.Ki -jf ^i^.-:i');in-ln -i n 1 help them work *vll 
together," or "When you den] vith (nursing) Clients, you've t;ot 
to understand tl r i L r viewpoint and trieir valuer and feelings ... 
t h "it's pa r t of y o • i r 1 la g no s i s . M 

What is significant here i.3 f first, the repeated pattern of 
change fron skepticism to -"insertions of value for "liberal 
education" experiences, on the part of students who remain 
primarily career- focused . Second, the pattern includes r 4 ot 
simply assertions., which might only be environmentally acceptable 
noises. Students make relationships between their concepts of 
learning- and their learning experiences, and give concrete 
explanations of how they see these kinds of learning as valuable 
to their careers, 'and to their personal life experiences. 

Closely allied to this pattern is the consistent importance 
of competence, or demonstrated ability, in the student's ongoing 
enterprise of preparing herself for career-role performance. The 
fact that the faculty have explicitly identified abilities within 
their disciplinary, subject ar.eas, and have linked " them to 
career- role, performance, seems clearly to provide students with 
the "missing pieces" to link classroom and workplace in their own, 
cognitive structures. 

At least as important, however, is the steady 
students 1 descriptions of feel in .*y, , am* 

certainty. This seems t at-. - ..-.-.j -iselv . itedly 

jw — to their - st.e.; .^-.accumulating experience of not only 
identifying but actually^ being- able to demonstrate these 
car.eer-valuable abilities. The areas of Communications and 
oocial Interaction are- earliest 'and m£>st frequently - , cited; 
perhaps since they involve areas that ar*e particularly 
problematic for the. young student just entering the college 
environment. 

Students al30 consistently broaden the settings in which they 
describe themselves using their abilities. As they progress, 
they ci te ^instances from work, family and other environments as 

-often as their in-class assessments. This indicates that 
cognitively they have made the transfer which they claim to have 
made experient ■ ally . Through exper. itial validation of the 

.competences, s ,l(V • . ! .s are able to construct a justification for 
liber- learning in which personal growth and effectiveness 
mediate between educational experience and concepts of 
professional role performance. 
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HOW DO COLLEGE OUTCOMES RELATE TO THE WORLD OF WORK? 



We used three approaches to examine the relationship between 
the outcomes shown by college students and the world of work. 
Two separate data sources result: (a) alumnae perceptions of the 
abilities involved in the workplace, and of the value of learning 
in their own evolving life goals (Mentkowski, Much 5 
Siencke-Holl, 1983), and (b) studies -of the abilities actually 
used by professionals in job situations (Mentkowski, DeBack, 
Bishop, Allen ft Blanton, 1980; Mehtkojeici , 0'3rien, McEachern <S 
^owler, 1982). 

The data from both sources suggest that graduates ' experience 
direct transferability Into the workplace of abilities learned in 
college/- and there are key abilities exercised by effective 
professionals which are both, conceptually and , statistically 
linked to college learning. 

Alumnae n - .ti ; 
S Wo r k , Lea m i n : - _ and C a r e er i ng 

The ;aary fin-: ig here is that graduates, while they are 

h-^hly su essful in ac sieving their immediate career oriented 
Is, continue to regard learning as a major value arid an 
nportant part of their lives" after college. They report that 
their work settings provide major demands and opportunities for 
continued learning, an important part of developing competence in 
the job role. At the same time, they,, describe it as an 
intrinsically rewarding activity which motivates career 
development, including job choice. 

To enable a cros3-sec tional comparison of graduating senior 
expectations with ^alumnae realizations, all 63 two-year alumnaein 
the clasd of 1978 who graduated from the outcome-centered 
curricn 1 um iainl on en ted in 1 973 were surveyed in Spring, 1980. 
Fifty-six .responded to the Alverno College Alumna Careering 
Questionnaire (Mentkowski h Bishop, 1980a), providing information 
on work and* career history, expectations- and satisfactions,, 
continued education % planned or achi.eved, and a variety of 
judgments about the value of college preparation for educational, 
personal and career goals. 

A parallel form (Mentkowski k Bishop, 1981 ) simultaneously 
surveyed. 63 of 58 graduating seniors for similar information 
except they were asked to anticipate career satisfactions. 
Thirty-two of these two-year alumnae had been interviewed with 
the Student Perspectives Interview (Mentkowski h Much, 1980b) at 
graduation and veve now invited for another indepth two to three 
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hour Uunna Perspectives Interview (Mentkowski i Much, 1980a). 
These ^2 women also contributed- a— questionnaire. 

Graduating 3tudent3 almost uniformly expeqt to work after 
college. Ninety-six percent of the alumnae surveyed actually did 
seek employment' upon getting their degrees. Ninety-two percent 
were successful, and 89 percent f ound^posi tions directly related 
to their majors. Since our background data also indicate that 
these women are more likely to obtain professional positions than 
their mothers, education seems clearly to function for them as an 
effective route to professional careers. Graduating seniors had 
higher career expectations than alumnae were able to> realize, in 
two years, but alumnae r3ted aspects of satisfaction with their 
fir-t positions and potential for advancement as above average. 
Alumnae also show a more positive attitude about their college 
lerning after two years than seniors express at the time of their 
graduation, although both* groups rated their college experience 
as above average on a majority pf items ( Mentko wski / Much & 
Gj encke-Holl, i 983 ) . 

Alumnae Transfer Abilities 

Given their generally positive attitude toward college 
preparation, how did their abilities carry over to performance? 
Alumnae viewed work not as justa job, but as a career that 
changes and .is developed through work experience. In analyzing 
the interviews for how graduates perceived learning on the job, 
we found two major categories of complex abilities that were 
equally imi- tant in managing their careers. Both younger and 
older women,- across all professional groups, emphasized 
interpersonal abilities learned i allege as important to their 
career performance ("You are more aware of your interaction 
skills, how you present yourself to people and how your 
interaction affects work relationships") and their continued 
learning. Alumnae also consistently cited reasoning 
abilities—using such terms as "analysis," "problem solving," 
"decision-making," "planning" and "organizational abilities." 

Alumnae Integrate and Adapt Abilities 

These abilities are often integrated and overlapping in 
practical role performance. They range from simply becoming 
familiar .with a new environment and new tasks ("Learning is a big 
part of what I do because when I started this job everything was 
new to me") to becoming an active learner in trying to carry out 
the role effectively ("I'm still learning what to use and I'm 
trying new things every year to see what's going to work"). 

Ideals for performance and accomplishment must be modified in 
the work sotting. Because of the environmental press, abilities 
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learned in college must be adapted ("I learned you have to 
introduce changes very slowly and gradually and to teach about 
the change before it happens"). New attitudes, beliefs, 
perspectives and elements of .self-concept or professional 
identity are a-equired as well ("My ability to compromise and be 
more tolerant and openminded has increased"). A young nurse who 
Kbb already made a k job . change summarizes the processes of 
adapting abilities to new environments ("You have to think more 
in terms of 'how do I do this in this si tuat ion" ... You are more 

consciously involved ;in what you are doing"). 

...... 

J : * " c 

Alumnae Experience Continued Learning 

As educators, we hope graduates do continue to have 
significant learning experiences after college. Abilities 
learned in college are an important stepping stone to effective 
performance ("What I learned most from Alverno and what's helping 
me most in my learning now is the whole process of learning, of 
starting and building on what you know, taking it from there"). 
Learning experiences are recognized and sought because "There s 
always a different situation that might come up." Another 
graduate comments that "Alverno ' taught me that I like to learn 
and that I am capable of learning." 

Continued learning is an intrinsic value which motivates 
career development including job choice. Graduates consistently 
speak of the importance of learning as "part of life," "part of 
my job" or "part of the person I am."' An older graduate in 
management talks of her career 'plans : in her current setting and 
adds, "If the learning starts tapering off... I would' consider 
going to another company. . .because I cann^ be stagnant in 
learning." A young nurse affirms: "To me, "Lying is learning." 

Alumnae Seek More Formal Learning 

In assessing commitment to more formal learning by graduating 
seniors, 36 percent expressed definite plans to continue their 
education after college, while another 57 percent indicated' a 
desire to .continue but did not specify a timeline. These 
expectations were more than realized by the group of alumnae two 
years out of school. Forty-one perpent did , complete additional 
education, and 56 percent expressed plans to acquire Additional 
formal education at a future date, showing a high commitment to 
continued learning- 
Alumnae Experience Competence 

Besides the abilities themselves and how they can be., used and 
adapted, one kind of* learning that becomes most critical to 
career' development and career management is the sense of 
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competence* The concept of competence is clearly important as 
they organize their career role performance an,! try to improve 
it. But the experience of competence is a key factor in career 
management and job cl~7ange. Graduates- viewed, work not. just as a 
job, but as k x -career that changes and develops through 
experience. Older women had a specific direction toward long 
range career goals; younger ones were more Immediately focused on 
developing competence in. their present job. But for all of them, 
career satisfaction is ' strongly related to their experience of 
competence on the job. 

Experiencing competence seemed to be a critical factor in 
whether she changed her job or career, and seemed to "carry over" 
from one job to another. "While graduates adapted to problems 
encountered in the first two years of work, including the 
traditional "reality shock" or d issillusionment experienced by 
most new graduates, the persistent feeling of not being able to 
perform in the job role, for whatever reason, led to change of 
jobs, if not career fields. Such changes were generally 
successful and appeared to re-establish the woman's feeling of 
professional competence, the basis for her self esceem. 

Abil ities JJs_ed_ by Pro f e s s i o a a . 1 s_ 

The major purpose of our studies of effective professional 
performance was to build a bridge to professionals in order to 
evaluate abilities most faculty would identify, and to create 
learning and assessment tools based on outstanding professionals' 
job performance interviews (Mentkowski, DeBack, Bishop, Allen & 
Blanton, 1980; Mentkowski, O'Brien, McEachern h Fowler, 1982). 
Eighty nurses from three health care settings (community, 
long-term care, acute care) and over 100 women managers and 
executives from more than 50 private corporations provided us 
'with job performance interviews, careering and professional 
development histories and ratings of critical competences for 
education, selection and outstanding performance. 

Competence Models of Effective Performance 
for Higher Education Programs 

More important, the competence model3 — particularly the one 
for management — suggest a sequence in the development of these 
abilities. For example, some personal maturity and intellectual 
abilities ' preceded the ' development of interpersonal and 
entrepreneurial abilities for managers. Helping behavior by 
nurses seemed basic to influencing clients to change, which 
formed the foundation for coaching clients to make their own 
changes toward better health. Since both models provide , 
behavioral descriptors, the model <;an be used to evaluate other 
programs in higher education as well as to develop more effective 
curricula and assessment criteria. 
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Both studies yielded models of the broad abilities that 
characterize effective on-the-job performance and showed a 
remarkable similarity to those identified by the faculty (Alverno 
College Nursing Faculty, 1979). The patterns of abilities that 
describe effective performance in two of the major professional 
areas for which we prepare graduates confirm the importance of 
focusing on interpersonal abilities (e.g., Development of Others, 
Coaching clients to take responsibility for their own health, or 
Influencing subordinates or clients by setting examples). 
Cognitive abilities represented in both competence models include 
logical thinking and using concepts to solve problems. Both 
studies also support the importance of Independence and Accurate 
Self-Assessment as integral to on-the-job performance. Other 
abilities that show personal maturity (Self-Control , Perceptual 
Objectivity, Stamina and Adaptability) crossed both professions 
a s we 1 1 . 



Clearly, those results suggest that outcomes developed by 
colleges must include attention to more than the knowledge 
component of abilities. And these abilities — which cross 
position level and even careers — can be abstracted by colleges 
and built into general education curricula. The abilities that 
were profession-specific (e.g., Entreprenurial Abilities or 
Helping) become the cornerstone for further development n 
particular majors. 

Abilities Develop Through Education 
and Experience 

Further, some abilities are developed more through education 
than experience on the job. Nurses with a bachelor 1 s degree were 
more likely to demonstrate Coaching, an ability that requires a 
complex form of helping the person ta change his or her behavior. 
Nurses with more experience were more likely to demonstrate 
Independence. Those abilities developed through experience on 
the job should >be part of learning experiences coordinated with 
off-campus work placement:*. 

While level of education was less related to performance for 
women managers, those who had completed a management training 
program showed more 3tamina and adaptability and more use of 
socialized power in dealing with subordinates. Tho3e who showed 
more rapid advancement in their company, and most likely a wider 
range of experience, demonstrated more Accurate Self- Assessment; 
and better developed Sol f-Presen tat Lon skills. 

Technical Ski. Lin An 1 Not Enough 

The f-iot, that pro fens iona 1 n in both studios demonstrated a 
wile range of complex abilities shown that graduates with 



functional or technical skills alone will not be effectively 
prepared to meet the demands of either nursing or management 
positions. For example, specialized knowledge did not play a 
critical or decisive role in the situations described by 
effective managers. Certainly specific training is needed for 
any entry level position, but for the person who plans a career 
in the two professional areas we researched, an education that 
prepares them for the future will include learning to integrate a 
number of abilities, to test them out in a range of actual work 
situations, and to critically appraise one's own performance. 



Both Perceptions and Performance Are Important 
Sources for Validating Outcomes 

In both studies, we researched abilities through performance 
interviews, and also asked professionals to judge a range of 
performance cha rac te ristics . Managers generally perform 
abilities they independently judge as characteristic of 
outstanding performers. T tfe found much less congruence between 
the performance o r nurses, and those characteristics of job 
performance nurses judged as critical for education, selection 
and descriptive of outstanding peers. The findings do allow us 
to identify those abilities that professionals don't demonstrate 
but identify as impor tant—like negotiating and networking in 
management—that signal abilities that should be part of the 
manager's repetoire. On the other hand, abilities like 
demonstrating self-control are more important for effective 
performance than the managers realize, and this finding suggests 
that curriculum objectives be tied to the study of actual 
performance. The assessment of competence is important to 
realizing long-term goals (Klemp, 1980). 



Adaptive Learning is Critical 

For both professional groups, performance of abilities is 
influenced by the context in which it occurs. For example, 
nurses in a community agency demonstrate more of the identified 
competences than do nurses in acute and long-term care settings. 
Managers from larger organizations demonstrate more of some 
competences, like Development of Others, Management of Croups and 
diagnostic Use of Concepts. In either case, demonstration of 
abilities reflects opportunity to do so. Larger organizations 
seem to provide more opportunity for women managers. Community 
health agencies provide more freedom for exercising a nurse's own 
professional expertise. Such influence by the work environment 
suggests that; adaptive learning is nri tical for adequate 
o n- the- job pe r f o rm an c e . 
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WHAT HAVE WE LEARNED? 



We believe we can show that complex abilities can be 
successfully integrated as a result of college experience 
stressing an outcome-centered approach to learning. These 
abilities can be identified, developed and used to achieve 
success in educational and work environments. We can also 
identify contributors to effective outcome-centered learning. 
These include instruction itself, as measured by the amount of 
student change on the external instruments, and the impact of 
education on the demonstrated abilities of professionals. 

Ou tc o mes Are Com pie x , jtol is tic 1 1 u nmn _A b i 1 i tie 3 

We researched outcomes through several different frameworks 
and measures. It is clear that definition and measurement of 
college outcomes needs to include a range of dimensions: 
cogni t ive/intellec tual process, affective/ socio-emotional 
process, perceptions, motivation and performance. The out comes 
studied by our battery of twelve external measures were 
differentiated into two separate factors at entrance to college 
and two years later, but were integrated by graduation. This 
suggests that educators may need to differentiate cognitive and 
affective aspects of abilities in order to teach for them. Most 
educators are struck by the difficulty of any attempt to separate 
these aspects. Yet attention to each dimension in turn may be 
necessary to enable students to integrate them later on. We need 
to study carefully just how this integration occurs, and what 
aspects of the learning process seem to develop this merging. 

It is also clear from our study of student performance on 
external measures that educators defining competences or 
abilities need to attend to individual differences in level of 
cognitive development and what implications this has for 
.lev eloping instruct ion. Vacuity who rated students on a set of 
performance characteristics neom to be tapping a level of 
cognitive develop' ent, since the ratings correlated with 
cognltive-developme level. Tf faculty are thus aware of 

students 1 developmental level along with the more tangible 

abilities involved in their day-to-day assessments we might 

infer that they also take developmental level into account in 
instructional planning and their interactions with students, to 
challenge and support their learning. 

Fu r t h e r , complex out c o in es n e e d to L n c 1 ud e a p e r f o r rn a n c e 
dimension. Learning to perform, to link knowledge to 
performance, is a dimension separate from the 

eogn i t ive- i ntel I r»<: tual one. This is borne ou t: by the fact that 
cogni. tive- developmental level correlates with the concrete 
experience/abstract conceptualisation dimension of learning 
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styles. Cognitive-developmental level doe3 not correlate with 
the reflective observation/active experimentation dimension. 
Educators have long sought an adequate learning theory that 
incorporates not only knowledge and cogni tive/intellec tual 
processes, but also the more practical learning that occurs when 
ideas are tested out in actual situations. This practical 
learning can be expected to transfer acros3 contexts to the world 
of work.' For students in our stud, learning to perform, to link 
knowledge to performance, enabled them to find reasons for 
learning in a variety of ways. They tried out the competences 
through application to professional performance and in their 
personal life. By doing so, they expe rientially validated the 
competences or abilities they were developing. The < ncept of 
"competence," which implies "knowledge and action, becomes a 
motivational link as well. Students began to see themselves a3 
competent. Thus, outcomes have a perceptual and motivational 
dimension that assist in their internalization and transfer. 
Values and motivation for performance have their roots in 
students' justification for learning as a stepping stone to a 
career and economic mobility. Along the way, continued learning, 
a liberal arts value, becomes part of the student's reasons for 
continuing in college. The student perceives herself as a 
^elf-directed learner, who seeks "well- round edness , " as well as 
* areer goals. 

Complex abilities, which include cognitive, affective, 
behavioral, motivational and perceptual components, do fit 
together and/or integrate to 3ome degree by graduation. This 
suggests that the abilities are hol istic, that is, that they 
involve the whole person. 



Outcom es Ar e Deve 1 oj^nejij^al 

Outcomes Develop as the Result of Instruction 

In this atudy, complex outcomes change over time, and arc 
related to performance in the learning process. Thus, they are 
developmental or teachable. We can link outcomes specifically to 
college instruction in at least two rays: (l) by the analysis of 
student change on the external instruments, and (2) by showing 
the impact of education on the demonstrated abilities of 
professionals. 3oth of these external sources validate the 
testimony of faculty who think they see students learning, 
external assessors who validate some of these abilities, and the 
testimony of students in confidential interviews who say they are 
learning them and whoso reports become more complex in describing 
their abilities in college, at work and in their personal lives. 



Just an student perceptions change, faculty ratings of 
pe rfo rmanee cha rac te ria t Los al so show students cha ng ing . 
further, student performance of faculty-dof Lned abilities of 
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Communications and Valuing are related to instruction. Students, 
by their own report, find the3e Communications "and Social 
Interaction abilities useful for functioning in personal and 
professional roles. 

On the other hand, there are other complex outcomes and 
competences where the link to performance in the learning process 
was less clear (e.g., Social Interaction). Older and younger 
students differ on some abilities and not on others at college 
entrance, and show some different patterns in developing them. 
How education interacts with experience to enable the student to 
build on informal learning outcomes is important in designing 
instruction to fit the adult learner. These results show that 
the outcomes or abilities are developmental, or teachable, and 
that we need to research other abilities to more carefully probe 
the dimensions of those abilities that can be linked to college 
1 earning. 

Outcomes Develop at Different Times 

^ually important is the time frame for development. There 
are differences in when these abilities develop during the 
college years. And as educators have always suspected, there is 
a difference between the general education experience and the 
later years when the 3tudent focuses on a major. Older and 
younger student;-* perform differently with respect to some 
cognitive-developmental patterns and abilities but not others. 
Further, abilities that may be differentiated during the first 
two years of college become integrated during the last two years, 
although how this happens is not clear to us now. The competence 
models developed from effective professionals show that abilities 
differ in complexity and sequence and suggest that the 
pedagogical order of abilities is important and can be 
identified. Professionals more likely develop some abilities 
exclusively on the job. But while experience may add to a 
student's ability to take advantage of college, some key 
abilities critical for effective work performance are clearly 
developed over time in long term formal learning experiences. 

Developmental Patterns Are Subtle and Complex 

When we look at the rate and quality of change, we note the 
kinds of 3ubtle and complex developmental patterns that will 
ultimately be of moat use to practitioners and to theorists. As 
we study students' developing perceptions, we see that students 
do value open-mindedness and self-direction and seem to 
demonstrate it increasingly. We would like to be able to account 
for how students actually do undergo the changes that they 
demonstrate in their interviews. It is our belief that 
communicating these subtle and complex patterns across 
disciplines, and to a wide range of faculty, is a prelude to 
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identifying the criteria for assessment of these abilities. We 
also think that being better able to define criteria for 
assessment will lead to Improved instruction, and consequently, 
improved educational validity of the learning process. Such 
efforts- will begin to link developmental theory and educational 
practice (Astin, 198*5) • 

There Are Individual Differences 
in Growth Patterns 

We have made some progress in identifying individual 
differences that account for how much particular students benefit 
from college. We then might be able to measure the3e 
determinants so faculty could get a better handle on who is 
changing in what kinds of ways, and be able to read the signals 
for change ' and transition. In doing so, we are moving towards 
the study more individual patterns of growth. Understanding 
individual differences is important to adequately serve the new 
student . 



Students progress through the assessment process with no 
noticeable deficit for the older student. The very structure of 
Weekend College (an alternate time frame which requires more 
independent learning) attended mostly by adult students presumes 
that the older adult can move at a more intense rate. Not only 
must she evidence this cognitively, but also in organizing 
multiple roles and responsibilities. The older student's life 
experience is not ignored, and there is no evidence of older 
students having any disadvantage from being away from formal 
academic work. In fact, there is a cognitive advantage that 
allows them not only to cope with the program but to cope with a 
concentrated program that's even more demanding. That 
presumption is borne out by students attending "Weekend College 
who alsc performed on external instruments; changes are not 
differentially less than the performance of students in the 
regular college time frame. However, our analyses of the 
external instruments show that age is an advantage in some area3 
initially, because the experience that it implies enables the 
student to take on this cognitive overload and deal with it 
successfully. And by the time they are more than two years 
along, the educational environment itself is a more likely 
determinant of learning. At the same time, older students begin 
at the same place as traditional age students in some areas, such 
as their understanding of classroom learning proce33C3 and roles. 



This difference between the older and younger adult shovra up 
after college. The experienced adult has more specific direction 
toward long-term career goals, in contrast to the younger 
graduate who is more focused on immediate ones. Both groups, 
however, report having to apply the name kinds of abilities to 
post-college work nettingn — interpersonal and reasoning 
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abilities. Thus, adults have an advantage in some areas but not 
in others. But it appears that college at least is capable of 
capitalizing on the differential abilities of the new learner. 

Another reason for this confidence on our part is that the 
college we have been studying has a traditional mission to serve 
working class students who are often first generation college 
students. ?or this student group, higher education can build on 
their particular strengths and background and enable them to 
continue capitalizing on it even when they've graduated. Women 
alunnae we studied show upward job mobility compared to their 
mothers. 

This study also show3 that the new student can be served when 
educators act on the .assumption that abilities learned interact 
with the student's value for learning. Values for education in 
this student group are linked to career outcomes. Indeed, a 
career-oriented rationale for college learning seems to describe 
today's student (Astin, 1982). The concern is that work-related 
rationales will bypass or shut out the traditional liberal arts 
values for lifelong, continued learning, and for benefiting 
others in society. The results from this study indicates that 
those fears are unfounded for "this student group. During 
college, values for personal growth and continued, lifelong 
learniag emerge. These values become linked to professional role 
performance, and to a perception of the self as a competent, 
self-directed learner. That these values for learning continue 
beyond college is evidenced by job changes made by alumnae who 
have inadequate 6 opportunities for new learning at work. 

Liberal art3 educators do, however, need to be aware of the 
values for learning of its student groups. Apparently the 
concept of competence and learning to perform are strategies that 
-an build a bridge from the practical values students bring with 
them to the more intangible values for lifelong, continued 
learning and professional role performance — where benefiting 
others is a primary, aim. Clearly, responding to new students 
means being as insightful as possible about the. reality of the 
learning experience for them and how they relate values for 
learning to their own goals for being in college. 

Col lege Outcomes Relate to Careering 
and Professional Performance 



Abilities Identified by Liberal Arts Educators 
are Demonstrated by Professionals 

The outcomes of college are generic, that is, they transfer 
to post-college settings. While level of education is linked to 
effective performance on the job, the abilities identified as 
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crucial to effective performance, like reasoning abilities,' 1 on 
the face of it, are similar to those identified by educators. In 
contrast, interpersonal abilities, long an expected result from 
informal learning alone, are critical to effective performance as 
evidenced in the professional's competences and the observations 
of working alumnae. Clusters of abilities carry forward from 
college to the world of work. While they must be integrated and 
adapted to the work place, they contribute to effective 
performance. Both competence models of professional performance 
show that professionals demonstrate a wide range of complex 
abilities. While the type of organization in which they are 
employed seems to influence the competences they perform, there 
are abilities that are generic, that transfer across setting and 
occupation. 



Abilities Need to be Adapted 

Our results strongly suggest that adaptation of abilities is 
such an important process for the graduate that college learning 
needs to specifically train for it. While college graduates will 
always face disillusionment and the conflict between realizing 
their ideals and making a living, how they deal with the 
challenge seems an important component to making the transition 
from college to work. Student values for learning to perform are 
realized in alumna motivation to adapt abilities to a range of 
contexts. Alumnae test out new ways of doing things to find out 
what will work. Learning how to adapt abilities involves ^ a 
process of applying judgment and abilities in action, getting 
feedback and adjusting accordingly. 



Learning Continues After College 

Learning to learn means discovering how to derive from an 
environment and experience what one need3 to know to adapt one's 
abilities. That the value for learning is interalized is shown 
not only because career satisfaction is built partly on 
opportunities for new learning but also because graduates go on 
to more schooling. Alverno students come to college for job 
preparation, ^rom their point of view, the college prepares them 
adequately, and they are almost all successful in finding the job 
they want after graduation. Over 40 percent of the alumnae we 
3tudied have continued formal learning two years past college, 
and another r j0 prcent expect to do so in the future. 



Abilities Learned in College 
Are Used by Alumnae 

There is a remarkable congruence between the abilities 
graduates say they use in the work place and those educators 
consider important outcomes of college. Interpersonal and 
reasoning abilities are both mentioned as necessary for coping 
successfully With a range of situations. 



Professionals-' ^perceptions of abilities descriptive of 
outstanding performers were congruent with demonstrated abilities 
cn the job for managers but not for nurses. Why this is the case 
is not^ clear, but it suggests our plan to assess the effective 
work performance of alumnae in addition to their perceptions is 
wise. The fact that alumnae have focused on developing abilities 
during college might make for more congruence, but this cannot be 
taken for granted. Since career variables like salary and status 
are not linked to effective performance for women in an emerging 
field like management, colleges seeking to validate their 
curriculum for women alumnae need to rely on performance as their 
indicator rather than measures of perceptions and self-report 
indicators. of career advancement. 



Competence is a Concept 
and an Experience 

College outcomes arid work are related very strongly by the 
notion of sel f-pe rceived competence. It is a cognitive organizer 
for learning both in college and at work. It is one of the most 
powerful experiential triggers for development according to 
students 1 testimony about how they manage their career changes, 
and career satisfactions. The mere act of identifying outcomes 
and giving people a chance to practice them has a powerful impact 
that carries from college to later life. Educators attempting 
outcome-centered education are in large part responding to the 
press for work usable education without sacrificing what 
education has traditionally meant. It seems fairly possible to 
take a liberal education and define it in terms of outcomes and 
make those outcomes expe r ienceable to stud ents , c reat ing an 
effective, lasting link between education and the world of work 
without having to sacrifice the value of those complex outcomes 
that motivate the liberal educator. The outcomes of liberal 
education can be identified, and when identified and experienced 
by the student, do persist. 

Enha ncing Feasibility 
Through Methodological Contr ibut ions 



Throughout the project, we pail particular attention to 
discussing methodological issues usually generated by large scale 
validation efforts. Several methodological contributions are 
discussed here because they point to the feasibility of either 
starting or continuing such work. 



lining Expert Judgment in 
Production Measures 

In i iberal arts colleges , expert judgment of complex 
abilities Is the primary mode of assessment. An English theme, 
History term paper, Eine Arts performance, student teaching, 



109 



117 



Nursing clinical, or Science lab all call for an extensive 
" productio n" by the student and complex judgment by an instructor 
on the extent to which the performance meets criteria. The 
closer the student performance i3 to abilities 3tudent3 will be 
demonstrating across work and personal roles, the more confidence 
the instructor has that the measure is valid. Yet many paper and 
pencil tests, that call for recognition alone are heavily used 
because of their efficiency in administration and scoring. With 
the advent of the computer, these measures are also more 
efficiently validated. 

In general, however, assessment of upper level work is often 
far too complex for such recognition tasks, and some disciplines 
in the arts and humanities are les3 likely to develop and use 
measures that rely on quantitative methods of assessment 
developed in the behavioral sciences. If criteria for judging 
are defined to give a picture of the abilities being assessed, 
faculty can more easily discuss common abilities that cros3 
disciplines and set the stage for reinforcing these 
abilities — like communications — across courses. And such 
measures become likely selections for a validator seeking to 
measure the outcomes of college. 

Thus, we used expert judgment in developing or choosing 
instruments for establishing program validity. Both arts and 
humanities, and behavioral sciences faculty are open to 
qualitative analyses of student responses, and expert judgment 
seems to mesh more with assessment strategies already in use. 
Faculty become more systematic and efficient expert judges very 
quickly, and are interested in specifying the basis for judgment 
and creating criteria. We have therefore adopted some measures, 
designed to be cross-disciplinary, as validation instruments. 
Some measures serve multiple purposes for diagnosing and/or 
c redentialing student performance, and also for validating the 
curriculum, adding to instrument efficiency. 

In addition to creating and validating measures they already 
U3e in the curriculum, faculty have been able to create complex 
new instruments and apply them with the validation team. Thus, 
faculty in an institution can, with professional help, do much of 
the work of creating cross-disciplinary, production measures of 
abilities, and also judge student performance. professional 
help, and al30 judge student performance. 



In this 3tudy we val idated a range of faculty-de3igned 
assessment techniques and a criteria and process for judging 
student performance on the Perry scheme. We tested out a variety 
of strategies. Current methodologien for validating 
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f acul ty-designed , generic ability instruments reflect a pattern 
analysis approach rather than score analysis, correlational 
analysis or an item analysis approach alone. These methods have 
implications for similar programs which are seeking new methods 
to establish construct as well a3 content validity of complex 
assessment instruments. 



Several criticisms of previous college-wide assessments like 
the Scholastic Aptitude Test include lack of relationship to 
later careering and professional performance. Grade point 
average, aptitude and achievement'-scores have not predicted later 
success (McClelland, 1973). We have therefore used a variety of 
new cognitive-developmental, learning style and abilities 
measures, performance interviews and inventories of 
professionals, as well as indepth interviews of student and 
alumnae perspectives to test out new methods of assessment. At 
the same time, we used some of the more traditional methods as a 
check on how results from newer asessment techniques compared. 
T 4e found that newer methods do take more time and involvement but 
are more efficient for other reasons. The descriptions of 
outcomes these measures yield stimulate more discussion by 
faculty, have more validity for performance after college, and so 
enable colleges to establish validity for particular professional 
areas. For example, by identifying competences that make for 
effective performance in the nursing profession across various 
contexts, we can build better incollege testing techniques and at 
the same time, contribute to revising state board examinations in 
nursing to make them more per form an ce- based . 

In general, cognitive-developmental measures, measures of 
learning styles, and some generic ability measures proved to be 
effective measures of change during college, and our studies of 
professionals* performance yielded a cadre of abilities that can 
serve a3 criteria for assessing the performance of alumnae. This 
will enable us to build assessment techniques for judging 
performance interviews of alumnae, a future goal. 



The press for accountability in colleges creates a need for 
evaluation and validation strategies. But assuming that such 
strategies can be applied without concern for the history and 
traditions of the liberal arts would only serve to alienate many 
faculty and administrators. Demanding that colleges now devote 
significant effort and resources to establishing validity in ways 
outlined by the behavioral sciences alone insures the failure of 
the approach. 



Testing Out New Measures of College Outcomes 



Defining Validity a3 a Concept 
for Higher Education 




Some liberal arts colleges have valid concerns about the 
press for accountability, and rejec. the assumption that 
establishing evaluation and validity are the most cogent response 
to such pressures. Fears that such attention to evaluation would 
mechanize, or otherwise destroy the primary values of the liberal 
arts have been expressed. If we attempt to identify and validate 
complex abilities, won't we end up with abilities that are 
mechanistic and so specific that a quick study can demonstrate 
them easily? Strategies 3een as advances in the behavioral 
sciences and educational research community may not be 
appropriate for liberal arts contexts. and the practice of 
program evaluation and validation of developmental outcomes needs 
new approaches. In any science, behavioral or otherwise, new 
paradigms are critical for solving new problems. 

Throughout our reports, we speak to the issue of adapting 
validation goals, strategies and designs to the particular goals, 
strategies and curricular plans of the faculty and involving 
faculty in carrying out validation efforts. Establishing 
validity means first identifying its meaning and use in a 
particular context. We do not suggest that a college incorporate 
our design or methods, but we hope colleges will find some of the 
strategies we used helpful to insure that their definition of 
validity and validation design builds on and is consistent with 
existing academic administrative structures and college-wide 
^oals. \e found validity best defined as a process that is 
developmental, illuminative and' diagnostic , and relates theory to 
practice and research to evaluation. Such a definition of 
validity suggests a validation design fitted to the context in 
wnich it is applied. The extent to which results from validation 
studies can be incorporated into an ongoing curriculum and used 
by faculty to'' improve it is the ultimate test of their validity. 
In. this liberal arts setting, faculty involvement was essential 
to meeting research objectives. 

Can a TAberal Arts College Accomplish Its Own 
Evaluation and Validation? 

During the past decade, responding to demands for 
accountability usually meant contracting with an outside 
evaluation consultant or agency who then developed and executed a 
design. Resources for such external evaluations are dwindling. 
And persons doing the research are absent when the real work of 
evaluation begins— impl ementing the results, we bui.t our own 
internal and external evaluation/validation mechanism, and then 
monies channelled into validation served as seed money to develop 
the abilities of college staff. While the evaluation staft 
fluctuates depending on availability of outside funds, there are 
consistent, evaluation and validation studies constantly ongoing 
and .supported by the college. 
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Ongoing, intra-insti tutional evaluation has raised the 
quality of other internal evaluation conducted in the college. 
Faculty outside the behavioral sciences are more willing to 
consider evaluation as part of curriculum development because 
they no longer bear all the responsibility for a task that has; 
its roots primarily outside their field. An expert staff isi also 
available to assist faculty with grants calling for evalua u ion, 
and faculty are moVe willing to enter into relationships with 
outside fund ing sources. Fund ing agencies are . more 1 ikely to 
provide funds for a project that has the mindset and demonstrated 
expertise for rigorous evaluation and validation. Responsibility 
for self-evaluation encourages close attention to exercising 
cbj.. .tive, analytical judgment, and to submitting plans *nd 
reports to outsiders for critique and review. 

Developing Participant Involv.ement Strategies 

One outcome of our efforts was developing strategies to 
involve students, alumnae, and professionals from the business 
and professional community. Methods of assessment often' involve 
using an unobtrusive measure where the participant is not sure 
what is being assessed. Tn our case, we needed to use measures 
that had face validity for a range of persons. We needed to 
generally -inform- them about our objectives so they would continue 
to participate in longitudinal research. But how could we 
guarantee our results would not just reflect a halo effect or the 
will'ingness of participants to "help" us by using as much of the 
acceptable jargon as possible? 

We did inform participants of the nature of our validation 
goals. But we also used a range of complex indicators and 
assessments ( indepth interviews ; cogni tive- developmental 
instruments which assess* growth over long periods of time) " to 
help ensure that outcomes were actually there. And we had to 
develop some data analysis methods to differentiate beginning 
from fuller understanding of the concepts we were assessing. At 
the same time, we met research standards for objective data 
collection and analysis. 

Using informed participants also served to meet other college 
goals . Creating relationships with alumnae, bui Id ing bridges to 
the professional and business community, and valuing students 
evaluation, helped enormously to establish the credibility of our 
college and its programs with these groups. 

Researching Women's Abilities 

Because Alverno is a women's college, the research reported 
here is on women participants . Women from ages 17 to 66 are 
dr^wn from student, alumnae and professional groups. Because of 
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the paucity of findings on women's abilities, results reported 
should be helpful to other educators and researchers who are 
attempting to understand women 1 s abilities and develop programs 
for the large nunbers of women returning to college. 



Disseminating Results to Ensure Implementation 

Research findings need to be disseminated to educators in a 
mode that has some practical , implications for educational 
..programming. The strategies used in this project include 
presentations at conferences and publications, but also to 
representatives of 150 colleges, universities and other puolic 
and private institutions who attended special information and 
workshop sessions at Alverno College. Research progress, initial 
findings and problems had to be presented in understandable 
language and linked tc practical issues important to other 
colleges attempting to improve their programs. Concurrently, 
progress and initial results were regularly disseminated to 
another 100 institutions by mail, through 25 presentations at 
conferences, to student research participants, and to faculty in 
a variety of settings and modes. Such dissemination strategies 
had the effect of constant questionning of the research 
methodology and identifying those aspects important to 
disseminate to educators. 



In addition, research findings could b&gin to be implemented 
through tryouts of various instructional and assessment 
strategies. Deutsch and Guinn introduced learning styles 
assessment as a regular part of new student seminars (Note 1; 
Mentkowski * Giencke-Holl , 1982); Loacker and Cromwell adapted 
criteria for judging performance on the Perry scheme to 
communications learning and assessment strategies (Note 3). 
Schall and Guinn used the Behavioral Event Interview competence 
assessment technique and a performance characteristics inventory 
in, a project to expand faculty awareness of abilities 
professionals use on the job. About 20 faculty interviewed 
another 130 professional in a number of professional areas during 
the summer (Schall Guian f> Note 4; Loacker 3 Schall, 1983). 
Student perceptions were used to improve career development 
3ervices (Fowler, Mentkowski % Schall, 1981). Faculty in natural 
sciences and technology designed investigative learning 
laboratory experiences and researched resulting student 
performance and perceptions (Truchan & Fowler, 1.980). Data from 
the longitudinal study of student change helped inform practice 
in the Office of Instructional Services (Neide3, Note 5). 
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SUMMARY 



Thi3 effort to enhance the quality, effectiveness and 
validity of education in the liberal arts has already contributed 
to several common objectives of college, their faculties, 
students and alumnae, the educational research and evaluation 
community, outcome-centered education, as well as our own 
college. 

Projects like this one and others can help colleges to take 
the initiative to define and demonstrate their outcomes to 
various constituencies who ask that such outcomes of college meet 
certain standards for use. Showing just how the more complex 
thinking and problem solving abilities show up at work, and how 
adaptability in learning on the job functions for the new 
graduate in the business community can make a difference to this 
segment of society who has often created their own educational 
technology rather than turning to colleges for help. Building a 
bridge to the business and professional communi ty v in ways that 
show we value their input in education — not just for ideals but 
practical abilities — c^n encourage them to join forces with 
educational institutions. Demonstrating that we are willing not 
only to identify outcomes as goals, but to deal with practical 
realities in making college work for students and for the 
business and professional communities we serve opens up a wealth 
of input, particularly for smaller colleges. The brain drain of 
professors from the research university to corporations can be 
reversed at the level of undergraduate education if we tap the 
expert judgment of their top personnel. 

Students and alumnae also benefit. Students benefit because 
they begin to feel that education is a process. Changes can and 
do occur, and students have input into program design and 
execution. Such a model sets up the expectation of themselves as 
change agents within the institution, and suggests a creative 
tension between the ideal and the real while they are still in 
college. While letting them in on the imperfect role of 
authority, it prepares them for the dynamic interplay between 
their own expectations for change and the conditions that are 
necessary for making changes. 

This is particularly of benefit to the cadre of new students 
who are most likely adults. Already part of the working world, 
they come to college with a more practical stance and expect more 
concrete benefits. They are under immediate pressure to show 
family and employer that the financial and time investment is of 
benefit at work and at home while they are still in college. 
This is more critical for women since many manage multiple roles. 
And the traditional age college student, who is currently more 
focused on . piactical career goals, will benefit from new 
strategies that build on prior formal and informal lean *.ng 
experiences. 
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Efforts like this one ars expected to more directly benefit 
faculty in making it more possible for them to improve 
instruction.' Most educators, pressed by the day to day 
frustrations and pressures of classroom instruction are open to 
identifying problems in teaching and looking for solutions. 
Having this helpful source, in addition to others, maintains and 
stimulates their work. When results from a cros3-college effort 
are available on a continuing 'basis, a common excitement and 
probing occurs. A. collaborative sense of purpose strengthens. 

Not to be overlooked is the benefit to interdisciplinary 
discourse of insight into student development and learning 
processes that can cross the barriers erected by the most 
independent department. Educators need both anecdotal and 
systematic results that describe ways in which students develop 
beyond one instructor's class, to ] if e after college. 



Thi3 model show3 how faculty and evaluators can work together 
with faculty's concerns driving the validation effort. Tt 
enables faculty to measure things they really care to change, 
instead of measuring outcomes .for which they are held accountable 
but that are not their own goals, toward which they are not about 
to change their teaching. It is also a model for devising really 
usable validation tools, because they come from the faculty's own 
goals and questions. That puts faculty in the position of being 
able to join with the administration and with the institution as 
a whole, in explaining to the rest of society what it is that 
education doe3. It enables the faculty to take the initiative 
and to regain the position of educating society about education, 
a task which has too Ion*., fallen into the hands of journalists. 



There are some contributions to outcome-centered education 
that also emerge from our work. First, we have felt confirmed in 
our decision some ten years ago to specify outcomes as a faculty 
and to develop curriculum through that common lens. We have 
certainly had more direct access to establishing the validity of 
our various outcomes and assessment process. The apparent 
success so far of our attempts to validate one faculty's 
outcome-centered approach to education suggests that 
outcome-centered education in education in general may indeed be 
a meaningful advance toward making college outcomes accountable. 



In doing so, we have been able to open the more subtle 
aspects of the learning process to critique and clarification. 
New theories of learning, particularly for adults, can emerge 
from this dialectic. Focusing on outcomes, and defining them as 
'complex processes, has enabled us to link learning to learning 
goals. Identifying abilities is an activity that helps students 
to cognitively structure and organise their own vision of the 
learning process in school, gives them a framework for 
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establishing the relevance of liberal education to their career, 
and helps them organise their careering after college. Finding 
that such abilities and processes transfer to their personal 
lives during college frees them to become more open to learning 
not directly related to a specific occupation, but to human 
growth. Tne effort to assess outcomes actively, as well as to 
identify them, gives the -student an important experiential sense 
of her own competence that seems to be a major catalyst in her 
development, both in school and at work after college. 

ft student's sense of her own proven competence becomes the 
organizing principle for her vision of her own growth and her 
strongest sense of proof. After college, it becomes a criterion 
for judging whether she is effectively managing her career, for 
judging whether she is satisfied in her job, for making job 
changes if necessary, or for staying were she is if she is 
satisfied. This experience is so powerful in college that after 
college it becomes her major criterion for assessing and managing 
her career. 



ft major reason for assessing the outcomes of college is to 
allow faculty to better accredit their students and to allow the 
institution to be more accreditable. But we have also found that 
the experience of identifying abilities and demonstrating them 
across multiple contexts i3 of tremendous learning benefit to the 
student, ft college that gives a student this experience is 
giving a student an advantage, whether or not outside groups 
would" identify those same abilities, or judge her effectiveness 
in the same way. 



Validation efforts in higher education also contribute to the 
educational research and evaluation community. Program 
evaluation as a discipline is new and is currently called on to 
provide technical assistance in the design of large validation 
studies in a wide variety of field settings. New technologies 
must be created to meet the demand, and methods that work in some 
educational research settings do not necessarily transfer to the 
cross-disciplinary atmosphere of a liberal arts college, or even 
to more technologically oriented universities where each 
discipline has its own well developed methodologies. It is not 
surprising, then, that we would have new validation definitions 
and strategies, improved methods for conducting educational 
evaluations, insights into disseminating and implementing 
results, and specific procedures for creating an atmosphere of 
fairness and respect for the input of our constituencies and 
informed involvement of our participants. 



We have made advances in the assessment of complex abilities 
including creating production measures where expert judges are 
trained ' to make qualitative judgments through objective, 
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analytical processes. Better ways to specify the criteria for 
judging complex abilities and for insuring their validity result. 



While we identify the values of validating outcomes for other 
colleges and constituencies, we trust the value to our own 
college comes through. We have made a commitment to continue our 
evaluation/validation process as part of the institution and a 
permanent component of the learning process. Our Office of 
Research and Evaluation is budgeted on a oermanent basis, 
although staff and breadth of activity may vary soinewhat 
depending on the strength of outside funding. 
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NEW DIRECTIONS 



There are several clear directions for future research and 
curricular applications. One is to probe further the meaning of 
the complex cognitive-developmental patterns, learning styles and 
personal maturity/ interpersonal and reasoning abilities we have 
begun to study. Improved criteria for assessment, improved 
instruction, and improved educational validity of the learning 
process will result. Researching such abilities can become a 
common task in higher education networks. We have begun to 
involve other colleges in one such effort to better define, 
instruct and assess for critical thinking (Cromwell and Allen, 
1982). 

It is now clear that abilities are complex processes where 
knowledge is only one component, and that the transfer of them to 
situations during college and afterward involves learning how to 
adapt abilities already learned to new situations and 
environments. Such learning on the job is much less guided and 
is predicated by "learning to learn" or "lifelong learning," 
familiar terms but not wel 1- researched concepts. The development 
of preferences for a range cf learning styles and change in 
cognitive intellectual growth measured by the Perry scheme may be 
the first indicator that learning to learn is developing for the 
student. But how do these beginning preferences become 
translated into sophisticated processes, in.o a "theory of 
action" for self-directed learning? 

We al30 need to make use of the patterns we have observed in 
students 1 developing abilities over time to research individual 
differences in ways of learning and in the expression of these 
complex abilities. An initial stage has concentrated on 
describing broad patterns of change. It does not speak to the 
question of who changes and why. For whom is college more 
effective? Who respond r ^tter to certain aspects of the 
curriculum, who does not: Retaining students who are not 
performing is critical to the survival of many public and private 
colleges. ' ' r ses of individual differences can specifically 
improve ins t rue 1 1 w rial strategies. We studied student change in a 
conservative research design to obtain a picture of the actual 
benefits of college. We also need to compare student entering 
abilities with those who did not persist, to see what abilities 
predic t staying in college. We have identified some determinants 
that account for how much ind ividual st udents benefit from 
college. These patterns need to be linked to instructional 
strategies . 

One practical way to build on prior efforts and to research 
these questions is to establish a system to continue to collect 
learning progress information based on developmental pattern 
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data. By monitoring students 1 progress through the learning 
process more carefully, and building our expectations on what we 
now understand about the development of abilities, we couli 
better pinpoint students who need more support and challenge, and 
intervene more quickly than longitudinal studies permit. 



Such an effort could also build on our increased confidence 
in the validity of expert judgment in identifying and assessing 
for abilities, and patterns in the way they develop. Qualitative 
expert judging processes can illuminate comlex student thought 
processes and behavior. Assessing complex competences, 
developmental level and learning styles is possible. 
Understanding such complexities is not beyond our grasp. We 
reinforce any efforts to further develop production type measures 
and judging processes of abilities that cross disciplines in the 
liberal arts. Many of the external measures we used need further 
work to be more useful to colleges attracting working class and 
minority students. We need to examine indepth the range of 
individual differences that occur in the normative patterns of 
change we have identified in this report. While our initial 
approach provides developmental norms for students at this 
college, and broadens the normative base for college students in 
general, it does not speak to the question of who does best in 
college and why. Retaining students who are not performing, 
particularly thoe who have not had strong educational 
backgrounds, is critical to the survival of many private and 
public ' 1 colleges. An analysis of these individual differences, 
and the abilities we studied, will be extremely helpful to other 
colleges. Further, we used a conservative research design to 
measure the effects of the learning process on students precisely 
to get a better picture of the actual benefits of college. We 
studied those who graduated from college across time. We need 
also to compare the entering abilities of those who did not 
graduate with those who did. 



Abilities demonstrated by professionals have good face 
validity with the outcomes educators usually identify. But these 
abilities, when described in a developmental sequence that takes 
into account the role of formal education and on-the-job 
experience, can help students in various occupational groups to 
be better prepared. Case studies and assessment criteria are one 
curricular application. Career advising based on professionals' 
careering histories are another. We need to continue to follow 
our alumnae and their developing performance abilities. A 
modified format of the job competence assessment performance 
interview would be a next step in studying their performance. 



Such efforts, to* research the meaning of abilities, to find 
ways to incorporate professional and career development abilities 
into the curriculum, to look for individual iifferences in the 
way in which they develop, and to create strategies for more 
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individualised instruction are important new directions. We also 
need to study how abilities are adapted and focused through 
continued learning, and describe learning to learn theoretically 
and practically. Creating a student progress information system 
that makes use of our current understanding of abilities, and our 
new assessment techniques can help to realize these goals. Our 
current interview research on individual patterns of learning and 
the determinants of change should also benefit faculty 
understanding of student growth. 

Fi na 1 ly , we need to con t i nue our e f f o r ts to d emons t ra t e a 
variety of validation strategies, models, methods and instrument 
designs. The fact that a liberal arts college has been 
successful in carrying out an elaborated effort, with the 
collaboration of a higher education and research network, is 
proof that colleges can develop their curriculum, do their own 
research and validation, and therefore continue to survive in 
American higher education. 



Meanwhile, our overall plan is to continue opening many of 
these issues and findings in more detail to the critique and 
comment of faculty in higher education, a process that will 
engage us and others in a renewal of interest in our chief 
concerns as educators. 



We have been excited while learning, using, and evaluating 
the concept of outcome-centered education. Alverno has been 
committed not only to -designing this kind of a curriculum, but 
also to designing an intensive measurement strategy to test out 
these ideas. The model presented here can be applied in other 
educational settings. It offers insights into new approaches in 
adult development which may have far reaching consequences in 
settings outside of higher education. In this way, we begin to 
insure that we develop abilities that truly last a lifetime. 
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Friedman, M., Mentkowski, M., Earley, M., Loacker, G., & Diez, M. Validating 
Assessment Techniques in an Outcome-Centered Liberal Arts Curriculum: 
Valuing and Communications Generic Instrument, 1980. 

Two studies test methodology for validating assessment techniques in a 
performance-based liberal arts curriculum. Alverno College has a 
system-wide performance based curriculum, with an assessment process 
that requires students to demonstrate incremental gains while progress- 
ing through six sequential levels in each of eight competences. The 
eight competences are integrated with the concepts in each discipline. 
Students are required to attain each competence level in sequence to 
demonstrate cummulative achievement. These two studies assess the 
effects of instruction on patterns of student response using instruments 
created to ensure cross-college credentialing on the same instruments. 
Both instruments are "generic/ 1 that is, general criteria are integrated 
with criteria specific to the way the ability appears in the discipline 
in which the instrument is used. Studies of two generic instruments, 
assessing level 4 ol the competences of Communications and Valuing 
are reported here. 

Twenty students performed on the generic Communications instrument after 
two years in college; another twenty performed upon entrance to college. 
They demonstrated abilities in four modes of communication: speaking, 
writing, listening and reading, providing data* on student performance 
across different modes of the same competence. The student is also 
asked to self-assess her performance in each mode on the same criteria 
on which she is judged by the assessor(s). Kleven students performed 
on the generic Valuing instrument after two years in college; another 
twenty performed upon entrance to college. Students demonstrated 
value and moral judgments and decision-making through written, oral 
and group decision-making nodes. Students also self-assess their 
per f o rniancu . 

In the Valuing study, the instruction group performed significantly 
better than the no instruction group. Data from the instruction 
group provided support for the validity of the cumulative hierarchical 
nature of the competence. The no instruction group did not show 
any consistent cumulative or sequential patterns. Overall, the 
instruction group demonstated clusters of relationships among scores 
on the criteria and the no instruction group appeared to perform in a 
randomly scattered manner, indicating effectiveness of instruction. 
In the Communications study, students with no ^instruction demonstrated 
a wider range of variability in performance as compared to the 
instruction group, who showed a less dispersed pattern. Student 
performance varies with the mode of communication. The instruction 
group performed significantly better particularly on the upper levels 
of the four communication modes. The different patterns of the inter- 
relationships of student performance across the four modes are seen 
in relation to the levels. Students who had instruction can better 
self-assess their performance. 

The study methodology reflects our current pattern analysis approach 
rather than using ;core analysis, correlational analysis or an toni 
analysis approach alone. The interpretation of the results and the 
methodology developed have implications for similar programs which 
are seeking, out new methods to establish construct as well as content 
validity of complex assessment techniques used in performance-based 
curricula in higher education. 
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Friedman, M., Mentkowski, M., Deutsch, B., Shovar, M.N., & Allen, Z, Validating 
Assessment Techniques in an Outcome-Centered Liberal Arts Curriculum: Social 
Interaction Generic Instrument, 1982. 

This report explores issues related to the validation of more 
nontradi t ional assessment techniques, and tests some ways such 
studies may proceed* We explore the appropriateness of various 
methods for validating a generic competence instrument that 
measures Social Interaction, a construct with little or no history 
as a teachable college outcome or measure. We compare the 
performance of 69 uninstructed students on entrance to college with 
that of 32 students who had two years of college instruction on 
each of the Social Interaction dimensions (Preparation, 
Demonstration, Self-Assessment and Leadership), and the specific 
dimension criteria. 

Results indicate similarities in performance between traditional 
age instructed students and mature uninstructed students. While 
this may be expected, it also indicates that group comparisons may 
not be an effective strategy for validating assessment techniques 
if the ability is one developed through prior informal as well as 
college learning. Despite our efforts to do so, we were not able 
to control for the myriad range of variables that are likely to 
affect the results. When performance of such an ability also 
interacts with a set of personal and ego development variables, 
separating out the specific effects of instruction that show 
significant differences through group comparison is not an 
effective strategy, especially given the small sample sizes 
generally avai lab le . 

However, some Social Interaction criteria did indeed separate the 
uninstructed students from the instructed students when we 
combined all students in a discriminant analysis. These criteria 
are more closely related to those aspects of Social Interaction 
that are learned as part of the more specific Social Interaction 
learning experiences. Thus, including students with a broad range 
of age and formal learning experience did lead to an effective 
strategy for identifying those Social Interaction behaviors that 
validate the construct. Clearly, the study of assessment techniques 
should not be limited to univariate methods; patterns of coherent 
group performance provide us with a more holistic picture of 
performance, particularly of Social Interaction, not well understood 
and measured compared to some other abilities like communications. 

The present study outlines a procedure by which the integration of 
information about compete ice construct, different group chnracter- 
istics and criteria evaluation contribute to an information base for 
instructional development, re-evaluation of competence definitions 
and revision of instrument criteria which measure these behaviors. 
The study helps to illuminate a key question in approaching the 
validation of any faculty designed instrument measuring important 
but not well defined abilities new to higher education instruction: 
What strategies are appropriate given where this instrument and 
construct are in their current development? 




Assessment Committee/Office of Research and Evaluation. Validating Assessment 
Techniques in an Outcome-Centered Liberal Arts Curriculum: Insights From the 
Evaluation and Revision Process, 1980. 

The Alverno College faculty has designed a curriculum and 
assessment process to assist students to develop and demonstrate 
ability in a variety of competences. Faculty, individually and 
as a group, design assessment instruments which then come under 
the scrutiny of other faculty in a continuous process of review 
and redefinition. This evaluation and revision process stimulates 
evaluation and revision of the instruments in a systematic way. 

Validating assessment instruments is an unusual goal for a college 
faculty to pursue. To validate means that concepts of the 
abilities or competences assessed and the means for doing so 
must be carefully thought out, subjected to rigorous reasoning, 
and constantly reviewed against student performance outcomes. 
This report summarizes questions, suggestions, concerns and insights 
generated from feedback sessions with faculty who submitted their 
instruments for a validation study. Sixteen instruments were 
identified by departments as ready to submit because faculty judged 
them sufficiently developed to evaluate. Three validation 
strategies worked best of those tried. One is pre- and post- 
instruction comparison which determines if changes in student 
performance can be attributed to the effects of instruction. 
A second is criteria evaluation, which involved the clarification, 
revision and refinement of criteria based on an analysis of 
student performance. A third is establishing the inter-rater 
reliability of assessor judgments, which enables a test of 
reliability as well as the development of instrument criteria. 
Criteria evaluation appears to be most helpful when the instrument 
is being evaluated and revised. Pre- ard post-instruction 
comparisons are used most effectively after faculty have judged 
the instrument as meeting most other instrument design guidelines. 
Inter-rater reliability s Indies are most useful when they are 
conducted concurrently with criteria evaluation. The validation 
studies that were synthesizeu for this report show that direct 
involvement of faculty in analyzing student performance data and 
probing validity questions generates a broad scope of validity 
issues . 
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Assessment Committee/Office of Research and Evaluation. Validating Assessment 

Techniques in an Outcome-Centered Liberal Arts Curriculum: Integrated Competence 
Seminar, 1982. 

The Integrated Competence Seminar assessment technique allows 
students to demonstrate integration and transfer of learning in 
three situations: Oral Presentation, liv-Basket Exercise and Croup 
Discussion. Assessors observe and evaluate performance against 
specified criteria, and give feedback to students on their perfor- 
mance. A behavioral criteria checklist permits evaluation of 
inter-rater reliability, and validation of the technique through' 
comparison of quantified assessor judgments with other student 
performance measures and a battery of external criterion measures 
.administered to students in a, longitudinal study of college 
outcomes (Mentkowski & Strait, 1983). 

Assessor judgments corre lated in the 70 ! s, The In-Basket Exercise 
vas the most difficult and the most valid in terms of correlation 
with measures of students' cognitive development and other generic 
abilities. Oral Presentation showed mixed results, and the Group 
Discussion correlated with other measures in opposite to the 
expected directions. When age, background and college program are 
controlled, there were no significant relationships between the 
three ICS tasks and other college performance measures, namely, 
number of credits accumulated and number of competence level units 
achieved. Thus, the In Basket had some performance validity, the 
Oral Presentation is equivocal, and the Group Discussion had 
relationships opposite to our expections. The finding on the Group 
Discussion supports our earlier findings with respect to a Social 
Interaction generic instrument. 

Generally, the effort revealed that the In-Basket exercise most 
• accurately measured abilities of- Analysis and Problem Solving. 
The Group Discussion, a measure of Social Interaction, worked less 
well. The study points to the importance of continuing to develop 
nontradi tional assessment techniques like Tn-Basket, and to 
revise the measure with particular attention to the links between 
Group Discussion criteria and the Social Interaction ability it 
represents. The Integrated Competence Seminal has since undergone 
extensive revisions by a group of faculty specializing in assessment 
design, based in part on the findings of this study. 
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Assessment Committee/Office of Research and Evaluation. Validating Assessment 
Techniques in an Outcome-Centered Liberal Arts Curriculum: Six Performance 
Characteristics Rating, 1983. 

The Six Performance Characteristics assessment technique provides a 
means for faculty to judge students in a systematic way over time on 
developmental characteristics which apply to their performance across 
disciplines and across competence areas identified as goals of liberal 
learning by Alverno faculty. Descriptions of six performance character- 
istics were prepared and first tested* by faculty on seniors graduating 
in the spring of 1978. The characteristics were integration, inde- 
pendence, creativity, self-awareness, commitment, and habituality. The 
characteristics were defined by sets of descriptors for the "Beginning 
Student," the "Developing Student," and the "Graduating Student." 
Pilot study results indicated some discriminating power (students 
graduating with honors were rated significantly higher than students 
graduating without honors). The following year all students in the 
college were rated to collect additional information on inter-rater 
reliability, the developmental character nf the ratings, and the extent 
to which the six characteristics were differentiated in ratings. 

Results from the first all-college administration provided evidence of 
acceptable inter-rater reliability, and supported the developmental 
character of the definitions through significant mean differences between 
classes. While the power of the technique to distinguish between 
students at different levels was demonstrated, it was found that all 
characteristics followed nearly identical patterns, raising further ques- 
tions concerning the differentiation between them. 

Si< Performance Characteristics ratings were conducted on all classes in 
1979, 1980, 1981 and 1982, as part of a comprehensive program validation 
which included other measures of student performance within the curric- 
ulum, and longitudinal assessments of student development and change using 
a batterv of external criterion measures (Mentkowski & Strait, 1983). It 
was confirmed with ratings from the longitudinal study sample of two con- 
secutive entering classes that a single factor accounted for 90% of the 
variance in ratings on each characteristic on three different occasions. 
Using' the single factor, it was found that students were rated at signif- 
icantly higher levels over time, corroborating the cross-sectional 
evidence for the developmental character of the procedure. The rating 
factor was not associated with other college performance measures in the 
longitudinal study when the influences of student background and program 
differences wore controlled. There was however evidence that ratings dis- 
criminated between students on academic probation and those who were not, 
irrespective of clas^ standing. 

Relationships between the Six Performance Characteristics factor and the 
measures of human potential revealed that the faculty we^e making judg- 
ments based on a general dimension associated with several external 
criterion measures of intellectual, ego, and moral development. The 
strongest pattern of associations was found with a measure of Perry's 
scheme of intellectual and ethical development during the college years 
(Perry, 1970, 1981). The Alverno faculty is continuing to work with the 
assessment technique, attempting to refine the definitions of several 
characteristics so that a more differentiated picture of student develop- 
ment may result. 
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Mentkowski, M., & Strait, M. A Longitudinal Study of Student Change in Cognitive 
Development and Generic Abilities in an Outcome-Centered Liberal Arts 
Curriculum, 1983. 

That students change in college is taken for granted. That 
students change as the result of performing in a particular 
curriculum is more difficult to show, and describing who changes 
and why, in relation to which complex abilities, is even more 
illusive. This longitudinal and cross-sectional study was 
designed to investigate three questions: Do students change 
in broad abilities indicative of human potential for cognitive- 
development, learning styles and other generic abilities? 
Can we attribute change to performance in a performance-based 
curriculum, rather than Co age, background factors and program 
characteristics? What are the underlying themes or patterns of 
change that could be used to assist curriculum developers in 
higher education concerned with responding to current frameworks 
in adult learning and development? 

Over 750 students participated in the longitudinal and cross- 
sectional studies by completing a battery of twelve instruments 
with developmental characteristics, and which employed both 
recognition and production tasks. The instruments were drawn 
principally from three sources: cognitive-developmental theory, 
experiential learning theory, and competence assessment designed 
to measure abilities which link those learned in college to 
professional performance afterwards. Students ranged in age from 
17 to 55; 200 formed a core group for the longitudinal study using 
a time series design with assessments at three times during college. 
Change occurred in varying degrees across the instrument set; 
some of this change could be attributed to performance in the 
learning process when age, background and program characteristics 
were controlled. Cognitive-developmental and learning style 
measures were better indicators of change than were the generic 
ability measures, suggesting that educators can measure development 
as an aim of higher education. As expected, recognition measures 
showed more change than the production measures. Initial 
performance at entrance to college was related to age for the 
cognitive-developmental measures, and to high school grades for the 
generic ability measures. While more change occurred during the 
first two vears (between the entrance assessment and the one two 
years later), the effects of the learning process on student change 
were more evident during the second two years (between the midpoint 
assessment and the one two years later near the end of college). 
Students appear to demonstrate two dimensions of cognitive develop 
ment, intellectual ability and socio-emotional maturity at entrance 
to college; these abilities are integrated by graduation. 

Implications for practice are that change is measureable, and that 
broad outcomes of college can be specified and assessed. Future 
interpretations of results specific to the several instruments and 
their interrelationships will more directly contribute to our 
understanding of the development of abilities learned in college. 
New outcome" measures have been tested, and the longitudinal data 
base of college learning is necessary to establish relationships 
between abilities learned in college and professional performance 
in followup studies of alumnae. 
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Much, N., & Mentkowski, M. Student Perspectives on. Liberal Learning at Aiverno 
College: Justifying Learning as Relevant to Performance in Personal and 
Professional Roles, 1982. : 

Approaches to the study of student outcomes at Aiverno include measuring 
performance in the curriculum, and student changes on measures indicative 
of human potential for cognitive development, learning styles and generic 
abilities (Mentkowski & Strait, 1983). This study explores student 
perspectives on learning as another valuable data source for validating 
abilities learned in college (Mentkowski & Doherty, 1933). How do students 
understand and justify learning outcomes? How do they understand liberal 
learning as relevant to performance in personal and professional rcles? 
Detailed analysis of interviews from 13 traditional age students at the end 
of their junior year were supported by systematic reading of 100 interviews 
from 37 women students interviewed longitudinally at the end of each college 
year. A qualitative methodology was selected that recognizes the subjective 
nature of the data and treats this "as a valuable source. Systematic proce- 
dures were devised for construction of content patterns representing student 
p erspectives on how they understand and justify learning and give meaning to 
day to day learning expediences. 

Two outstanding patterns consistent with curricular emphasis a^.d student 
orientation appear. First, students express a career centered rationale 
for college education. Learning is justified primarily in terms of its 
relevance to practicing a particular career after college. Second is a 
heavy emphasis on learning "how- to-do" things; learning is or ought to be 
useful. Students regard the learning process as concerned with teaching 
them how to perform and apply what they know. The meaningf ulness of day to 
day learning experiences is predicated upon perceived relevance of these 
experiences to professional performance. While students express dissatis- 
faction with learning experiences for which they cannot find career relevance, 
they succeed in developing a justificatory rationale for assimilating all 
kinds of learning including "wel Iroundedness , " a variety of discipline 
content areas and the competences, to the idea of professional role 
performance. For these students, the competences are central to the 
structuring of learning to perform; "use" or "application" of learning 
refers to the competences. Other kinds of substantive knowledge, observa- 
tions, ideas, concepts, theories and so on, are assimilated to the competences 
which structure learning to perform, and, are linked to role performance. 
Competences offer ways of looking at things, ways of understanding, ways 
to be aware of what is important. Students experience the competences as 
meaningful and useful and anticipate their application to the work setting. 
For competences Communications and Social Interaction, for example, students 
report feelings of increased mastery, control and certainty in three areas 
that students regard as important and which are often problematic for 
young women: interpersonal relations, identity and personal choice. The 
competences support student's perceptions of being more in control and more 
effective in common everyday social and work settings, including those 
encountered in off-campus experiential learning settings and personal life. 
Through experiential validation of the competences, students are able to 
construct a justification for liberal learning in which personal growth 
and effectiveness mediate between educational experience and concepts of 
professional role performance. 
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Mentkowski, M., Much, N., & Giencke-Hcll, L. Careering After College: Perspectives 
on Lifelong Learning and Career Development, 1983. 

This initial study of alumnae from Alverno's outcome-centered curriculum 
asked four questions: How are alumnae learning to learn at work, and do 
they describe lifelong learning? What abilities and processes enable 
transfer of learning to professional performance and careering after 
college? What are alumnae perspectives on careering and professional 
development? How do the expectations of students and realizations of 
alumnae compare? We conducted followup interviews with 32 alumnae, 
and administered a questionnaire to 56; 63 seniors also completed the 
questionnaire. 

Interview analysis indicated that continuation of learning is a positive 
value, is part of developing competence in the job role, and is valued . 
as intrinsically rewarding, which motivates career development and job 
choice. Learning on the job is based on abilities, including those 
learned in college. Complex abilities especially important for new job 
roles are interpersonal abilities learned in college. They were strongly 
emphasized among both younger ar.d older women and across all professional 
groups, as an important foundation for both performance and continued^ 
learning. Reasoning abilities such as analysis, problem solving, decision 
making, planning and organization also transfer to work. These abilities 
are integrated and overlapping in practical role performance. Learning on 
the job, apart from additional technical learning, involves further devel- 
opment of major abilities and their integration and adaptation in relation 
to work contexts. Learning how to adapt abilities involves a process of 
applying judgment and abilities in action, getting feedback and adjusting 
accordingly. Learning to learn means discovering how to derive from an ^ 
environment and experience what one needs to know to adapt one's abilities. 
Most women viewed work through some concept of careering, looking beyond 
the present job to a professional future. Professional ideals were impor- 
tant in relating to work. Older women had a specific direction to long 
range career goals; younger women were more immediately focused on devel- 
opment of competence in their present jobs. Career satisfaction was 
strongly related to experiencing competence on the job. Satisfaction 
with ability r:o do a job well is fundamental for careering. A feeling of 
persistent inadequate performance of the job role led to change of jobs 
or career. Such changes re-established a feeling of professional compe- 
tence. Work satisfaction involved job enjoyment, a sense of relaxation 
and being comfortable with work; and progress. All women had strategies 
for career progress, but older women had more complex and long range 
career strategies than younger women, who focused more on excellence now. 

The cross-sectional questionnaire study found that seniors expect to work 
after college; 96 percent of alumnae sought work, 92 percent succeeded, 
89 percent found work directly related to their major. These women had 
more professional positions than their mothers. Seniors had higher career 
expectations than alumnae were able to realize after two years, but 
alumnae rated satisfaction with a first position and potential for advance- 
ment as above average. Alumnae show more positive attitudes toward college 
learning after two years than seniors; both rated it above average. Forty- 
one percent of alumnae reported additional education; 56 percent said they 
planned more. Alumnae attribute more importance to educational goals than 
graduating seniors; both said they achieved their important career and 
personal goals. Older alumnae view analysis and self-directed learnii 
as more important than do other groups. Potential for advancement is 
powerful in determining career satisfaction. 
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Mentkowski, M., DeBack, V., Bishop, J., Allen, Z., & Blanton, B. Developing a 
Professional Competence Model for Nursing Education, 1980. 

The major purpose of this study was to create a generic competence 
model for effective nursing performance. The major outcome is a 
codebook describing nine generic abilities. The competences were 
derived after an intensive qualitative analysis of performance inter- 
views from 80 outstanding and good nurses in which nurses discussed 
what they actually did in situations that led to effective and 
ineffective outcomes. A peer nomination questionnaire yielded out- 
standing and good groupings of nurses; a background questionnaire 
provided information on education and experience. Nurses were 
employed in a long-term care setting, an acute care setting and' 
a community health agency. 

Nurses perform a great deal of Helping, a competence which fits with 
the more traditional role of the nurse. Fut they also perform 
Independence, Influencing and Coaching to a large degree, and they 
perform Conceptualizing. These competences describe today's nurse 
as an active, Influential professional who demonstrates independence 
and analytical thinking in her role. More of these active competences 
were demonstrated in the community health agency than in the acute care 
agency; the acute care agency and the long-term care agency segm to 
have a more structured environment with regard to roles and tasks. 
Nurses in a more structured situation may not demonstrate some of 
these abilities to a greater degree because of the demands of the 
sett ing . 

The more experienced or more educated nurse is likely to demonstrate 
more Conceptualizing, less negative Conceptualizing, more Ego Strength, 
and more Independence, Influencing and Coaching. These competences 
taken together seem to have an underlying copponent — an active,, 
thinking, influential style where the nurse also strives to assist the 
client to take on more responsibility for his or her own care. Some 
of these abilities appear more in the community agency, an agency we 
beli eve is likely to be more supportive of these competences, where 
more educated nurses are employed, and where nurses are likely to 
have more role autonomy. 

# 

This study contributes to efforts by nursing associations and 
educational programs to assess effective nurse competences. In this 
study., nurse educators and nurse practitioners were, able to cooperate 
in a common effort to develop a competence model that can improve 
nursing education. The 350 situations described by the nurses in 
the performance interviews can also serve to improve case study 
and other instructional and assessment materials. Nursing 
curriculum needs to built on the performance abilities of effective 
nurses . 
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Mentkowski, M., O'Brien, K., McEachern, W„ & Fowler, D. Developing a Professional 
Competence Model for Management Education, 1982. 

This study identifies abilities or competences that ensure effective managerial 
performance and sequences them to create a model of effective managerial per 
formance. Performance, perceptions and careering and professional development 
of 103 women managers and executives from 53 Milwaukee private corporations 
are described and related using a recently developed performance measurement 
system. Three outcomes result: a competence model of effective managena 
performance for improving management programs; a pool. of over 500 behavioral 
examples set within particular contexts that can be used in instruction and 
assessment; and better advice for women students seeking examples of 
careering and professional development and how it. relates to effective 
performance in the managerial role. 

No one competence dominates the performance of these managers. They demon- 
strate abilities across the broad spectrum of interpersonal, intellectual, 
entrepreneurial and socio-emotional abilities. Women managers demonstrated, 
intellectual and entrepreneurial abilities to the same degree as they 
demonstrated interpersonal abilities. Educators creating sequential manage- 
ment curricula and managers planning their own professional development can 
benefit by knowing whether some competences are prerequisites for others. 
Several factor, cluster and path analyses were performed. Competences are 
in the main independent of each other but some are best learned in sequence. 
A manager's ability to initiate rests on intellectual skills; ability to get 
the job done rests on people skills. Underlying these is self-assessment, 
the ability to learn from one's experience. 

Abilities effective managers judge as critical to outstanding performance are 
generally the ones thev perform in day to day situations. Two abilities, 
important to outstanding performance according to managers and that were not 
performed often in this study are using networking and negotiating win-win 
situations. Demonstrating self-control and positive regard for others 
abilities demonstrated often, are apparently more critical to effective man 
agerial performance than managers judge them to be. 

Implications for management education are that programs teach and assess for 
a range of complex abilities. Traditional management education has Reused 
on developing particular technical skills yet specialized knowledge did 
not play a critical or decisive role in the situations described by these 
effective managers. Education that prepares for the future will include 
learning to integrate abilities, to test them out in a range of work 
situations and to critically appraise one's own performance. Both work y 
environment and job function affect the extent to which these abilities are 
demonstrated; this suggests that adaptability of one's abilities is critical 
for effective performance. There are., however, a common set of broad 
competences educators can expect will generalize across situations^ 
contexts. Abilities on whicRthe Alverno program is built mesh with those 
demonstrated by effective managers. The study provides a cadre of inter- 
view material for building realistic and relevant instructional e ^^" 
ences, a model for sequencing competences, and insights into careering for 
structuring career development activities. 
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Mentkowski, -M., Moeser, M., & Strait, M, Using the Perry Scheme of Intellectual and Ethical 
Development as a College Outcomes Measure: A Process and Criteria for Judging Student 
Performance. Vols. I & II, 1983. 1 

This study describes use of the Perry scheme of intellectual and ethical develop- 
ment as a broad measure of growth in college in an outcome-centered curriculum. 
Issues that arise in applying cognitive-developmental measures to validate a 
learning process in relation to students 1 projected- potential for development 
during college are discussed, as are those questions educators raise in applying 
the Perry scheme when teaching. The study contributes to understanding, using 
and 'measuring the scheme, examines how colleges can demonstrate change as a 
result of curricula, improves adequacy of judgments made in relation to the 
scheme, and identifies issues educators need to consider in using the scheme 
appropriately. 

The report's major focus is description of the criteria and process that yields 
judgments of student performance relative to the Perry scheme. " The criteria 
(descriptive statements) and judgment process, together with a set of . 46 examples 
showing how the criteria are applied independently by assessors and through con- 
sensus, should assist other persons to analyze student performance relative to 
the scheme-. Reliability of the process for assessment and validity of the 
criteria and the instrument stimuli and mode are examined in relation to assessor 
decision-making and judgment, and student performance on essays. The study under 
scores the importance of continuing to research expert, judgment as a technique 
for assessing student performance in college. 

Parts of -Volume I and assessor training materials in Volume II can serve as a 
training and rating manual. Volume I contains the process for judging student 
performance on the Perry scheme, the Alverno criteria used in the judging process 
and documents steps taken to establish 1) reliability and validity of the judging 
process and criteria, and (2) validity of the Alverno criteria in relation to 
their use by assessors. Data reported contribute to establishing face, psycho- 
metric, criterion group, longitudinal, convergent/divergent, and educational 
validity of the Measure of Intellectual Development (MID). This measure, based 
on Knefelkamp and Widick's work, has a history of research and use to which this 
report contributes. 

Persons new to the Perry scheme, with little if any background in developmental 
psychology and theories of assessment did learn to rate essays at satisfactory 
levels of inter-judge agreement prior to consensus. Agreement prior to consensus- 
increased during the training sessions from 57% to 65% to 78%, which we believe 
resulted from concurrent improvements in four phases of* criteria, development . 
Inter-judge agreement on final rating of almost 3000 essays was 76% prior to 
con 'nsus, a percent reached by a new assessor trained in the judging process. 
Int-i-judge agreement with 'an expert external assessor was 67% prior to con- 
sensus. Analyses of almost 20,000 assessor judgments showed that the themes 
of the developmental scheme were found useful in judging essays. Some criteria 
were used more than others. Criteria of a general and specific nature were 
equally useful. Criteria from position 2 "What to Learn," position 3 "How to 
Learn" and position 4 "How to Think" were used most. Criteria that describe 
aspects of a stable position are distinguished from those that describe the 
dynamics of transition between positions. Thus, the criteria describe the | 
evolution of student change. Generic criteria are distinguished from those 
specific to essay type, and suggest applicability across other essay types or 
performance modes like the interview. Kssay specific criteria show how the 
content of an essay interacts with underlying structures in development. 
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Volume I also describes results from a five-year longitudinal study of student 
development- on the Perry scheme (Mentkowski & Strait, 1983), including relation- 
ships to other cognitive-developmental measures (Kohlberg, Rest, Loevinger, Piaget) 
Participants were 750 women aged 17-55. Applying the method and criteria, we 
found that the measure shows definite change in both cross-sectional and longi- 
tudinal studies. The criteria and process did work to describe differences 
between students and student change over time. The patterns of change, however, 
are the intriguing results. Each of the three essays, Best Class, Decision 
and Career shows change. Bul the change is not always straightforward. When 
development occurs depends on the area. Rate of development is related to 
age for decision-making and career understanding at entrance to college, but 
not for students 1 understanding of classroom learning processes and roles. 
But after two years, older students have made more immediate progress in 
understanding concepts such as learning through multiple ways, learning, from 
peers, and becoming independent il one's own learning. Formal learning experi- 
ences are necessary for enhanced understanding of these concepts. Student 
change on any, of the three areas of development is not related to high school 
grade average when students enter college, nor does it account for change 
during college. Students change on the Perry scheme, and development is 
differential depending on the area of development. 

These results illuminate the way students change in college, and examine the 
issue of the contribution to student development by the college experience 
for both traditional and non-traditional aged students. This study points 
to the need for careful translations between any theoretical model of 
development and its adaptation for program evaluation, instruction and 
assessment, and further theory building. 
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We disseminated progress and outcomes of the research 
reported in this overview and nummary at various points with 
several strategies. Altogether, we ( 1 ) made 27 presentations at 
state, national, and international conferences, and to 
professional groups in the Milwaukee community; (2) contributed 
to or created 10 publications that brought requests for more 
information; (*?) prepared six progress reports that identify some 
of the problems and issues we encountered in carrying out the 
research objectives*; (4) created seven reports for students, 
alumnae, and professional participants; (5) disseminated 
progress , procedures and results to the Alver no faculty, 
administration and trustees, and several college committees, 
departments and divisions; (6) made 22 presentations at Alverno 
College Workshops and Visitation Days, and distributed materials, 
where we had opportunities for discussion of several issues 
described in this overview with representatives from 148 
institutions.; and (7) mailed materials to persons at 102 
institutions (for a total of 250 institutions)- These 
dissemination strategies are listed below. 



• Presentations at state, national, and 
international conferenc es 

• Community group presentations 

• Publications 

• Progress reports to the National 
Institute of Education 

• Reports to student, alumna, and 
pro fessional pa r tic i pants 

• Reports to faculty, trustees, and 
advisory councils 

• D- S3emination to representatives of 
in3ti tut ions who visited Alverno 

• Dissemination through mailed materials 
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Presentation at State, National, 
and International Conferences 

Throughout the grant period, we presented the rationale, 
progress and preliminary results at state, national, and 
international meetings. Participants at these conferences 
included researchers, persons in business and industry, and 
college teachers and administrators. 

Mentkowski, M. Can the concept of human development supply a 
unifying purpose for higher education? Presentation at the 
National Dissemination Conference, Memphis State University, 
Memphis, TN, June 1933. 

Men tko wski , M . Is the re 1 ife after college? EstaM ishing the 

validi ty of college-le arned abilit ies for later careering and 
profes sional performance. Presentation at the Eleventh 
International Congress of the Assessment Center Method, 
Williamsburg, VA, June 1983- 

Mentkowski, M. , # Strait, M. A 1 o n g i t u din a 1 study of change in 

cognitive developm ent and gen eric abilities injan outcome-centered 
liberal arts c urriculum. Paper presented at the meeting of the 
American Educational Research Association, Montreal , Canada, 
April 1983. 

Mentkowski, M. , & Strait, M. Using the Pe rry scheme of inte llectu al 
and ethical deve lopme nt as a co_llege outcomes measure: A process 
and cr iteria for jud g ing student performance. Paper presented" at the 
meeting of the American .Educational Research Association, Montreal, 
Canada, April 1983. 

Mentkowski, M. Student dev elopment on th e Perry scheme. Presentation 
at the annual meeting of the Association for Moral Education, 
Minneapolis, MN, November 1982. 

Mentko wski , M. Current uses of experien tial learning theory at 
Alverno College. Presentation at the Brain and Learning Styles 
Conference , Chicago , IL, October, 1982* 

Strait, M. A study of co ll ege outcomes. Presentation to the Illinois 
and Wisconsin Association of Registrars and Admissions Officers, 
Rockford, TL, October 1982. 

Mentkowski, M. Issues in progra m evaluat ion . Workshop at the Second 
Conference on General Education at the Inter American University of 
Puerto Rico, San Juan, Puerto Rico, September 1982. 

Mentkowski, -M., & McEachern, W. Developing a professional competence 
model for mana gement education . Presentation to the Tenth 
International Congress of the Assessment Center Method, Pittsburgh, 
PA, June 1982. 
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Mentkowski , M. Us i ng t he Pe rjr^3^ch£inj^_o f J-JL^AL^PA"-"^ J*iH J iQlk "- "- . 
development hs a colleV^'out comes measure. Pap'er presented at the 
^h.Tn>~anI~/nTthVt :,i " Pe rfy "cbnf"" " " : e f Augsburg- College, Minnenpol is , 
MN, -June 1981 . 

Lo never G . , A Mentkowski, M. J_____J_UsJ^ 

assessment -enter methodology at _Alyerno. Presentation to the Eighth 
Tn"tel^ro"naT"Co"n7re3"s of the "Assessment Center Method, Toronto, Canada, 
June 1980. 

Friedman M. Validating change in_student outcomes . Presentation to the 
Wisconsin Al^oTiation for Toll eg late Registrars and Admissions Officers, 
Cheboygan, WT, October 1980. 

Mentkowski, M. , DeBack, V., Bishop, J., Allen, Z. , % Blanton, B. 

Developing a pro fessional _competence j^eWbr iursyig_^<^a^ioni. 
PiTpef1^orrted"at the meeting of "the American Educational Research 
Association, Boston, April 1980. 

Friedman 1. * Mentkowski, M. \^Wation _of J^essment Jroh^ues_in.iiS. 

outcome-centered liberal arts eurr iculum:_ J^ij^lJi^J^tj^. _ Paper 
presented Tt The" mTeTing 'oT "the Tmer ic an Educational Research Association, 
Boston, April 1980. 

Mentkowski, M. The design and asjjsjmjmtj^n 
' goal is the d^Jlo7mTntTf_ val-dng. Paper presented at the meeting of the 
feocTa"Uo^ _ "-'r"Mo"ral Education", Philadelphia, P A , November 1979. 

Mentkowski, M. Research implications and results from a study of 

learning styles and professional competences. In A. Wutzdorff (Chair , 
Learning to learn at work: _Ca.se atudj_,_irnj£e^^^^^ 

1^1^^" ""l^^ the meeting of the Council for 

thf"Ad7a"nfi^t:ent of Experiential Learning, St. Paul, MN, April 1979- 

Mentvowski M. A researched jsy^atiojwn^ 

learned in coTTege for later "success. ^aper presented at the Sixth 
J^^^ol^i^on^e-sr^k^s^ Center Method, White Sulfer Springs, 

WV, June 1978. 



Com munit y Group Presj^ntj^ions 

Another strategy for -dissemination calls for presentations 
and distribution of materials to community professional groups. 
Most of these groups consist of representatives of various 
professions. The following list is indicative of the range ol 
groups to whom we disseminated information and materials: 
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Presentations of the nursing study by Vivien DeBack, 
Nurs ing Chai rpe rson , to Milwaukee pro fess ional comtnun i ty 
groups : 

• Greater Milwaukee Area Nursing Service, 1980 
© Nursing Education Administrators Group, 1980 

• Cursing Administrators of the Visiting 
Nurses Assoc iation , 1 980 

• Sigma Theta Tau, a national, nurses 
honor society, 1980 

Presentations of the studies of alumnae and professional 
managers and nurses presented by Marcia Mentkowski to 
the following groups: 

• Law Auxiliary of Wisconsin, June 19S0 

o Association for Women Lawyers, 
December 1 981 

• Inner-Group Council, a group of 
pro fess ional women , June 1 982 

Presentation of results from the management study by 
Marcia Mentkowski and Kathleen O'Brien were made to: 

o Professional Dimensions, a group of 
professional women, including management 
study participants, February 1983 

Presentations of alumnae study of the integration of 
career and family by Marcia Mentkowski: 

• "Work and family: How can 1 do both?" Alverno 
College Telesi3 series: Bui lding on our 
exp e rience: Women talking wi th worn en. 1 982 , 1 983 . 




The following publ ic at ions include a range of mat e rials 
disseminated through a variety of ways: 

Student examples. In Alverno College Faculty, Assessmen t at Alverno 
College . Milwaukee , WI: Alverno Produc tions, 1 979 . 

Jennifer examples. In M. Earley, M. Mentkowski # J. Schafer, Valuing 
at Alverno: The Valuing Process in Liberal Ed uca tion. Milwaukee, 
Alverno Productions, 1980. 




Publications 
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"The Alverno Valuing Program: Jennifer Tells Impact of Program." 
Alverno Today, Winter, 1900, pp. 4-6. 

"Alverno Receives NIK Orunt." Alverno Today, Winter, 1977. 

"Office of Research and Evaluation Reports Findings." Faculty Newsletter, 
Alverno College, June 198^. 

"Alverno Research Identifies Nurses Abilities." For ward , 12 , 1980, 
pp. 49-50. 

"Best Nurses: Strong, Caring." The Milwaukee Journal, Sunday, 
June 15, 1981 ■ 

Heehinger, F. "Women's Colleges Going Strong." The Mil waukee Journal , 
Sunday, May 3, 1901 , p. 10. 

Mentkowski, M. Creating a "mindset" for evaluating a liberal arts 
curriculum where valuing is a major outcome. In L. Kuhmerker, 
M. Mentkovoki, & V. L. Erickson (Eds.), Evaluat ing moral development 
and evaluating ec^uc^a tjjoj^^ 

SchnectadyT ~NYT "Character Research Press, 1980. 

Valuing Competence Division, Alverno College. Underst anding the jt^ent^ 
pe rceptions o t Jl^ A^^E. 1 -^ Ji^^r^ab U i t__y ^_ _ Jjlil^i e _^p_ Jl^A^ j^JlLt^' ' 
through her"~cb liege years. Milwaukee, WI: Alverno Productions, 1980. 
(Videotape". M."llentkow3ki and M. Riederer dramatized this excerpt from 
Va 1 u i ng at Al ve r no :_ The _Va AA^i 1 !^ ^— .^ iberal Education. ) 



Progress Reports to the 
Nation al InsFiTut e_6f ^Education 

Mentkowski, M. , * Doherty, A. -C€£eeri^ 

the vali d ity of abili tie s JLe. arn_ ed JLn_ jiolle^e JJor Jate. r J^u cjmjss . 
¥Trsl~pr"o"gress report submit ted to the "National Institute of Education, 
February 21 , 1973. 

Mentkowski, M. , * Doherty, A. Careering jf t^r_c o U eg^ ej^ _ Jk t * b *M 

the validity of abil ities learned in coflegejj pjlJ^^^uj^ess. Second 
progress~Tepprt submitted " to the National' Institute of Education, 
July 15, 1978. 

Mentkowski, M. , £ Doherty, A. Careerin g after college: Establishing 
the validity of abilities learned in c ollege for later success. 
Year- end "pro, g resi s TeTpo r t submitted to the National Institute of 
Education, January 30, 1979. 

Mentkowski, M. , * Doherty, A. Careerin^a£te r_c !0 .11 eg e : Es tablishing 
the validity qf_ alu ies^M^ 

Mid-year - pFogress report submitted to the National Institute of 
Education, July 30, 1979- 
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Mentkowski, M. , 4 Doherty, A. Careerin g after college: Establishing 

the valid ity of abilities learned in college for later success. Second 
year-end progress report submitted to the National Institute of 
Education, January 30, 19 BO. 

Mentkowski, M., & Doherty, A. Careering after college; Establishing 

the validity of abiliti es learned in college for later success. Final 
progress report submitted to the Nationaf Institute of Education, 
September 30, 1980. 



Reports to Student, Alumna 
and p ro f e ssional Par tic ipa n t s 

Alumnae and participants in the professional studies in 
nursing and management and alumnae studies were mailed reports. 
Names of institutions are not listed to preserve confidentiality. 

Mentkowski, M. , DeBack, V., Bishop, J. , Allen, Z., S Blanton, B. 

De v eloping a pro f e s s iona 1 co m p e t e n c e model _f or nurs in g ed up _a ti o n . 
Milwaukee, WI: Alverno Pro due t ions , 1980. 

Mentkowski, M. , O'Brien, K., McEachern, W., & Fowler, D. Developin g a 
professional competence model for management ed ucatio n. Final report 
s urn ma ry~foV~pa~Fici parTts , Milwaukee , WI: Alverno Productions, 1983. 

Mentkowski, M. , Much, N. , & Giencke-Holl , L. Careerin g after college. 
Progress report to participants in a longitudinal study of college 
outcomes. Milwaukee, WI: Alverno Productions, 1903. 

Communications to students took the form o f oral 
presentations and letters throughout their participation in the 
study. In addition, we provided students with four written 
reports (in Msntkoreki A Strait, 1983, Appendix II). 

Mentkowski , M. Cha nges i n student pr ofil es on the Learning Styl e 
Invento ry. First report to participants in a longitudinal study 
of coll . Je outcomes. Milwaukee, WI: Alverno Productions, 1981. 

Mentkowski , M. Understand in g the d evelopment of thinking in __co^lleg_e. 
Second report to participants in a longitudinal study of college 
outcomes. Milwaukee, WI: Alverno Productions, 1981. 

Mentkowski, M. Some que s tions and an swers about ^ev^aluation studies. 
Third report to participants in a longitudinal study of college 
outcomes. Milwaukee, WI: Alverno Productions, 1979. 

Mentkowski, M. , & Fowler, D. Learning to learn . a t_ work Students 9 

alumnae and other professionals. Fourth report to participants in a 
1 ong i t u d inaT^st u dy~ of college outcomes. Milwaukee, WI: Alverno 
Productions, 1981 . 
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Repo rtg to Fa cu 1 t y , Tr us tees , 
and Advisory Counc ils 

Communications by the principal investigators to faculty 
about the rationale for the study, progress reports and results 
were made through oral presentations, memos, and copies of 
materials sent to students (Mentkowski 4 Strait, 198^, Appendix 
II). Presentations by the Director of Research and Evaluation 
highl ighting particular aspects of the work were made to the 
corporate faculty at the semester end institutes, and to new 
faculty each year who were given an overview of the study during 
New Faculty Orientation. Seven such presentations were made to 
the total faculty on the purpose, rationale and progress of the 
research. Thirteen more specific reports were made to various 
departments or committees in the college. The Director assisted 
Alverno's Career Development staff to instruct faculty in the use 
of the behav io ral even t interviewing techn ique for a summer, 1 982 
project to gather information about job abilities as they relate 
to careers. 



Two presentations were made to the Board of Trustees; the 
first dealt with the results of studies of student perceptions 
(Fall - 1977); the second reported on the purpose, rationale and 
progress of the study of managers (Spring 1980). 



There were several reports to advisory councils. They 
include reports to the Evaluation Advisory Council: October 
1978, February 1979, November 1979, June 1980, April 1981, April 
1982, and March 1983; and reports to the Management Advisory 
Council: June 1979, and September 1979. 



Dissem ination to Representatives of Institutions 
Who Vis i ted Alverno 



Visitation Days at Alverno 

An effective form of dissemination to persons outside Alverno 
was to representatives from 148 colleges, universities and other 
organizations who visited Alverno to attend workshops. A review 
of the list of institutions includes colleges and universities, 
corporations, and other private and public institutions. Many of 
these institutions sent a number of representatives to the 
specialized workshops for college teachers in assessment and 
valuing, and to Visitation Days. We believe this is an 
indication that these institutions are interested in improving 
practice in higher education and are willing to make a long-range 
commitment. 
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Visitation Day is a one-day session for persons interested in 
an overview of outcome-centered learning and assessment, 
attendees receive a one-hour presentation from the Director of 
Research and Evaluation on research results and outcomes 
supported by the National Institute of Education, The Assessment 
Workshop for College Teachers and the Valuing Workshop for 
College Teachers are one-week workshops . In the first , the 
Director presents an overview and summary; in the second, 
insights from research efforts are introduced as they apply to 
various area3 under discussion. Materials \are distributed to 
attendees and participants also may select from available 
reports. Our policy wa3 to initiate and maintain a network of 
relationships in the research and higher education community 
helps us adapt methods and develop instruments and procedures to 
meet the research objectives. p art of this network was created 
by the contacts made through early dissemination of our efforts. 

There were 30 institutions in Wisconsin that participated, 
which includes 14 of the institutions in higher education in this 
state. One hundred thirteen institutions and organizations 
participated at the national level, and six at the international 
level. It was these presentations with questions and discussion 
that most clearly focused our work, and was the most effective 
strategy for dissemination. With this final report, we expect to 
reach a wider variety of audiences who are concerned and 
committed to the validation and evaluation of higher education 
programs. The 148 institutions to whom we disseminated research 
outcomes and materials at the state, national, and international 
level follows. 
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Institutions deceiving Office of Resennh 
■ind Evaluation Presentations and 
Materials at Visitation Days 
an 1 Workshops 

V)77 to 



V/_i_nc_o nsin 

All is Chalners 
Milwaukee , W iscon3in 

Appleton Electric Company 
South Milwaukee, Wisconsin 

A pc hd i oc ese of Mi 1 wauk ee 
M LI waukee , Wisconsin 

Audubon Middle School 
M i L wa uke e , W i s co r. i i n 

Carthage College 
Kenosha, Wisconsin 

Edgewood College 
Madison, Wisconsin 

Genu Church 

Mil waukee , W isconsin 

Immaculate Heart of Mary Parish {?.) 
We3t Allis, Wisconsin 

Inroads, Inc. 

Mi 1 wa uke e , Wisconsin 

3. C. Johnson # Son, Inc. 
Racine, Wisconsin 

Marian College of Fond du Lac 
Fond du Lac, Wisconsin 

Marquette University (?) 
Milwaukee, Wisconsin 

Medical College of Wisconsin 
Milwaukee, Wisconsin 

Milwaukee Area Technical College 
Mil waukee , Wisconsin 
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Milwaukee School of Engineering (2) 
Mil waukee , Wisconsin 

Milwaukee Urban League 
Mil waukee , Wisconsin 

Pius XI High School 
M i L wa uke e , Wis co n s i n 

St. Alphonsus School 
Greendale, Wisconsin 

St. Frederick Parish 
Cudahy, Wisconsin 

St. Gregory Parish 
Mil waukee , Wisconsin 

St. Joseph Convent 
Milwaukee, Wisconsin 

United Community Center 
Mil waukee , Wisconsin 

University of Wisconsin - Green Bay 
Green Bay, Wisconsin 

University of Wisconsin - Madison 
Mad isc n , Wisconsin 

University of Wisconsin - Milwaukee (3) 
Milwaukee, Wisconsin 

University of Wisconsin - 03hkosh (2) 
Oshkosh , Wisconsin 

University of Wisconsin - Parks Id e 
Kenosha, Wisconsin 

University of Wisconsin - 

Platteville (2) ' % 
Platteville, Wisconsin 

\ 

University of Wisconsin - Whitewater 
Whi te water, Wisconsin 

Wisconsin State Council on Economic 

Educa !:ion 
Milwaukee , Wisconsin 
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A C- om s i 1 ta f . l o m ^)'i:>o r t i am fo r 

0 r ga n i z - 1 t ioin] TV » v e 1 o pm e a t 
Vernon Hills, Illinois 

A 1 .43 -: t ?h r. L f i c Un i. v e r s i t y 
Aneho ruge , Mask-* 

A I v e nil a Hi S c h o o 1 ' - ■ ) 
C hioago , II 1 mo is 

Anna Marie {College 
?4Xtoa t Massachusetts 

A:;:;i)' ,: nVion of Catholic Colleges 

and 'In i vo rs L t ios 
Wash ; ng ton , D. C . 

Bali win-Vial lace College ( "5 ) 
Berea, Ohio 

Barat Sol lege (?) 
Vile-* "orest , II Lino is 

Parry College 

Miami Shores, Florida 

Bay ie Noc Community College 
Escanaba „ Michigan 

Bellevue College (?) 
Bellevue, Nebraska 

Bellmont College 
Nashville, Tennessee 

Bendix Corporation 
South field , Michigan 

Bethel College 

St. p .aul, Minnesota 

Brigham Young University 
Prove, Utah 

Oarlow College 
Pittsburgh, Pennsylvania 

Cedar Cre3t College 
Ml en town , p ennsylvania 

Chapman College 
Grunge , Oal if ornia 
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Clayton Junior College 
Morrow, Georgia 

College IV, Grand Valley :Uu^> 

Colleges 
Allendale, Mi c hi g an 

College of Lake County 
G raysl ake , II 1 inois 

College of New Rochelle 
New Rochelle, -New York 

College of St. Benedict (3) 
St. Joseph, Minnesota 

College of St. Mary (2) 
Omaha , Nebraska 

College of St. Scnolastioa (2) 
Duluth , Minnesota 

College of Saint Theresa 
VI inona , Mi nn eso t a 

The Cooper Union, Cooper Square (2) 
New York, New York 

CreLghton University (2) 
Omaha, N e b ra ska. 

C. W. Post College (2) 
Greenvale, New York 

Cuy^hog'-i Oommmity College (2) 
Parma, Ohio 

Delgado College 

New Orleans, Louisiana 

Be n 1.3 o n Un i ve rs i ty 
Granville, Ohio 

DePaul University 
Chicago, Illinois 

Dominican High School 
Omaha , Neurdsku 

Donnelly Coll e^e 

Kansas City,' Kansas ^ 

Elizabeth town College (2) 
Elizabeth town, Pennsylvania 

El Paso Community College 
El Paso, Texas 

Empire State College 
Old Westbury, New York 
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V 1 am i n g ^ a 1 a b ow Un i v e r 3 i. t y 
?ahl4quah, Oklahoma 

Florida A A M University 
Ta 1. 1 aha s 3 o e , ■ . F 1 0 r L < \ a 

Florida State University (2) 
Tallahassee, Florida 

Franklin University 
Columbus, Ohio 

Governors State College (2) 
Park Forest South, Illinois 

Harrls-Stowe College (4) 
St. Louis, Missouri 

Holy Name Co.llege 
Oakland, California 

Housa tonic Community College 
Bridgeport, Connecticut 

Hudson Valley Community College 
Troy, New York 

Illinois Institute of Technology 
Chicago, Illinois 

Illinois State University (?) 
No rmal , T 1 lino L 3 

Indiana Institute of Technology 
Fort Wayne,. Indiana 

Iowa Lakes Community College 
Estherville, Iowa 

John Brown University 
Siioam Springs, Arkansas 

Joliet Junior College 
Joliet, Illinois 

Kamehameha Schools (2) 
Honolulu, Hawaii 

Kapiolani Community College (2)" 
Honolulu, Hawaii 

Kentucky State University 
Frankfurt, Kentucky 
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Kirkhof College (2) 

Grand Valley State Colleges 
Al lend ale , Michigan 

Loyola Un iv ersi ty t 
Chicago, II lino Is 

Kirk wood Communi ty ' College 
Cedar Rapids, Iowa 

Lesley College (,?) 
Cambridge, Massachusetts 

Mar9 Hi LI College 

Mars Hill, North Carolina 

Mary College (7) 
Bismark, North Dakota 

Marywood College 
Scran ton, Pf-nnsylvan ia 

McKendree College 
Lebanon, Illinois 

Medgar Evers College 
Brooklyn, New York 

Memphis State University 
Memphis , Tennessee 

Mercer University in Atlanta 
Atlanta, Georgia 

Metropolitan College (3) 
Minneapolis , Minnesota 

Metropolitan State College (4) 
St . Paul , Minnesota 

Miami-Dade Community College (2 
Miami, Florida 

Michigan State University (2), 
"Justin Morrill College 
East Lansing, Michigan 

Mid-Plains Community College 
North Platte, Nebraska 

Minneapolis Community College 
Minneapol is , Minnesota 

Mount Marty College 
Yankton, South Dakota 

Nazareth College of Rochester 
Jew Yo HC'New York 



Office of Catholic Education 
Chicago , II iinois 

"lew York Society for Ethical 

Culture Schools (2) 
New York, New York 



North Adams State College (2) 
North Adams, Massachusetts 

Northeastern Illinois University 
Chicago , Illinois 

Northeastern University (4) 
Boston, Massachusetts 

Northwestern Univei^si ty 
Evans ton , Il linois 

Northwest Regional Educational 
, Laboratory 
Portland , . Oregon 

Nova University. 

Fort Lauderdale, Florida 

Otterbein College 
Westerville, Ohio 

Our Lady of Angels College 
Aston , Pennsylvania 

Our Lady of the Lake University 
San Antonio, Texas 

Park College 
Parkville , Missouri 

Pennsylvania State University 
University Park, Pennsylvania ' 

Rhode Island College (2) 
Providence, Rhode Island 

Rock Valley College 
Rockford , Illinois 



St.. Louis University School 

of Business (2 ) 
St * Louis , , Missouri 

St. Mary of Celle Parish 
Be r wy n , Illinois 

St. Mary's College of Maryland 
St. Mary's City, Maryland 

St. Xavier College 
Chicago, Illinois 

Trenton State College 
Trenton, New Jersey 

Trinity Christian College (2) 
-Palos Heights, Illinois 

Trinity College 
Washington, D.C. 

University of Evansville (2) 
Evans v ill e, Indiana 

c 

University of Minnesota 

School of Dentistry (2) 
Minneapol is , Minnoso ta 

University of Oklahoma 
Norman , Oklahoma 

Ur sinus? College 

Coll eg ev ill e, Pennsylvania 

Voorhees College (3) 
Denmark, South Carolina 

Waldorf- College 
Forest City , Iowa 

Walter Sundling Jr. High School 
Palatine , Illinois 

Washington International- Colleg 
Washington, D.C. 

West Oahu College (2) 
Aihea , Ha mi i 
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Spartanburg Technical College 
Spartanburg, South Carolina 

St* John's University 
Collegeville , Minnesota 

St. Leo College 
St, Leo, Florida 



Wharton County Junior College 
Wharton, Texas 

William Rainey Harper College 
Palatine, Illinois 

Wright Institute (2) 
Berkel ey, California 



Inter n ati o n $ l_ 

Brook Universi ty 
St. Catharines 
Ontario, Canada 

Inter-American University of Puerto Rico 
San Juan, D uerto Rico 

Sheridan College of Applied Arts and Technology (?) 
Oakville, Ontario , Canada 

Southwest London College, 

Center for Higher Business Studies 
London, Hngland 

University of Puerto Rico 
San Juan, r aerto Rico 

University of Puerto Rieu at Rio Piedras 
rjuyriabo, Puerto Rico 



Pis sem i n a t i on Jh r ojigh jM nil ed_ JfatejJ-^X 3 - 

Porsons from a range of institutions have written to us for 
further information, and we have responded by sending material s 
wherever possible that related to their efforts. 



In September, 4'v00 copies of Valuing at Alv erno: The 

Valuing Process in LiJl^ 1 ! Education TRnrloy, Mentkowaki 
Sc*ha f e"r , 1 0 > 557," ~whi c h c o n ta fn s e x to n s 1 7 e r e ferenoos to the NT hi 
funded research to validate Alverno's curriculum, were mailed to 
values educators and to academic deans in higher education across 
t. h° vmntry. 

The following institutions and representative departments 

received mut.'ria! s« 



165 



Institutions Receiving Requested 
Office of Research and Evaluation Materials 



1077 to 1985 



Wisconsin 

Department of Public Instruction 
Mad ison. Wisconsin 

The Journal Company 
Milwaukee, 'Wisconsin 

Department of Psychology 
Marquette University 
Mil waukee , Wisconsin 

r> t • Lukes Hospital 
' Mil waukee , Wi sco na L a 

Director of Hospital Education 
St. Michael Hospital 
Mi L waukee , Vlisconai n 

University of Wisconsin - Oshkosh 
.Oshkosh, Wisconsin 

University of Wisconsin - Superior 
3ii peri or, Wisconsin 



national 



Bureau of Study Counsel 
Ha pvard llniversi ty 
Cambridge, Ma s s a c hu s e 1 1 s 



Association of Pathol ic Colleges 

and l In i vi'rs i t, i »■»,? 
Wash i n^ ton , ^. C. 



Center for Moral ftduoa i, i on 
Ha rvard Un Wersi ty 
Cambridge, Massaohuso t f - 



Career Info nn a t ion r>y stem 
Kutfonn f Oregon 



C o n t r* a 1 M i o h i f > an Uni v e r : u. t y 
Moun t l) l on-ian t , Mi o h i flan 



Division of Programs in Education 
Program for Gifted Youth 
Hunter College 
'low York, Now York 



CoUe/;o of .?t. Honed i- t 
.H I; . Joseph , Mi nneso ta 



Internal ional Publ ic Po 1 ioy Hesearch 

Co rporat ion 
Mo [»ean , Vi r/*.i.n i a 



Nur s i a/^ Do pa r tin on t 

Co! 1 *w> M >t. Ronod i : t 



ERLC 



College of St, Benedict 
St, Joseph, Minnesota 

Associate Dean 

Delaware County Community College 
M ed i a , Pe nns y 1 v an i a 

The Edu-Caring Foundation 
Resources for Human Development 
A rdmore , Pennsylvania 

The Ethical Culture School 
New York, New York 

Forest Service 
Washington, D. C. 

George Meany Center for Labor 

studies, Inc. 
Tripartite Program for 

Apprentice sh i n a n d A s so c i a t e 

Decree in Labor Studies 
S i 1 v e r S pr i n g » Ma r y 1 a n 1 

Ci e o rge W a sh i n g to n Un i v e r s L t y 
Washington , T). C. 



ASM Schwartz College of Pharmacy 

and Health Sciences 
Long Island University 
Brooklyn, New York 

Dern for Acad em ic Af fai rs 

The North Carolina School of Science 

and Mathematics 
Durham , No rth Caro lina 

National Center for Research in 

Voca t ional Educat ion 
Ohio State University 
Columbia , Ohio 

Coordinator of Continuing Education 

for Nurses 
Oregon Institute of Technology 
Klamath Falls , Oregon 

Saint John's University 
Coll eg evil 1 e, Minnesota 

Director, Counseling Center 
Saint Peter's College 
■Jersey City, New Jersey 



Community Outreach Options for RN's 

Department of Nursing 

S an Jo se State Un i v e r s L ty 

San Jose, California 

Dean of Academic Planning 
S o n om a S t a t e Un i v e r s 1 1 y 
Rohnert Park , California 

Swa r thmo re Co 11 ege 

S wa r t h in o r e , r> o n n s y 1. v an i. - 1 

D i v i s i o n o f Ed uc a t L o rial F o u n J a t L o n s 
School of Kducat ion 
Sy r a ou r e U n i. v e r : \i t y 
S y r ' a c u s o , N e w Y o r k 

U n i v e r s i t y M o die a 1 C e n t r 
Un i v » 1 rs L ty o f Ca 1 i f or n i n - Da v i a 
F - un i 1 y Nu r s e p r a e t i t i i ') no r P r og ram 
.Sac ram en to , Ca !. i f orni a 



Graduate Institute of Education 
Washington University ■ 
St . Louis , Mi ssouri 

The Wright Institute 
Berkeley, Cal ifornia 

Teaching Re sea rch 

A Division of the Oregon State System 
of Higher Education 
Monmouth , Oregon 

Do pa rtrnen t o f Pro fess ional. Regulation 
Tal lahassee, Florida 

Department of Education and Psychology 
University of Massachusetts 
Am h e rs t , Ma s s a c hu s e 1 1 s 

Interinnti tut ional Programs 
I o wa R eg en t a Un 1 v e r s i t i. t e s 
Iowa C i ty , To wa 



Counsel i ng Cent" v 

■ In i vers i ty of Maryland Department of Higher Education 

College Dark, Maryland University of Kentucky 

ho x i ng to n , Ken tue k.y 
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Department of Psychology 
University of Maryland 
College Park, Maryland 

Experiential Learning Program 
Ho rribake Li b ra ry 
University of Maryland 
College Park, Maryland 

Graduate School of Education 
University of Utah 
Salt Lake City, Utah 



Ciaremount Graduate School 
CI a remount , California 

School of Medicine and Dentistry 
Division of Medical Education 
The University of Rochester 
Rochester, New Yo rk 

Center for the Teaching Professions 
Northwestern University 
Evans ton, II lino is 

Abt Associates, Inc. 
Cambridge, Massachusetts 



School of Education 

The University of Michigan 

Ann Arbor, Michigan 

Office of Institutional Research 
Moraine Valley Community College 
Palos Hills, Illinois 

CCC Mastery Learning Project 
Chicago, Illinois 

Psychological and Social 

Foundations Department 
College of Education 
University of South Florida 
Tampa, Florida 

Department of Educational 
Research and Development 
Rochester Institute of Technology 
Rochester, New York 

National Evaluation Systems, Inc. 
Amhe rs t , Massnehuse t ts 

Extended Learning Institute 

No rthorn Vi rgi nia Commun i ty Co 1 1 ego 

Annan dale, Virginia 

Bronx Community College 
City University of New York 
Bronx, New York 

Department of Curriculum ! ind 

Inst rue t ion 
II lino is State Universi ty 
Normal , Til. in a is 



Teachers College 
Columbia University 
New York, New York 

Department of Health Education 

St. Louis University Medical Center 

St . Louis , Missouri 

School of Nursing 
Texas Tech University 
Health Sciences Center 
Lubbock, Texas 

University of Minnesota 
Minneapol is , Mi nneso ta 

Project for Research in Nursing 
Adelphi University 
Garden City, New York 

Office of Educational. Development 
University of Alabama 
B i r m i n g ham , A 1 ab am a 

The American Registry of Radiological 

Techno log is ts 
Mi nnoapol is , Mi nneso ti 

•fount Aioysius Junior College 
C resson , p ennsyl van i a 

New York State Education Department 
Albany, New York 

School of Allied Health Professions 
Northern Illinois University 
DeKal b , II. I inois 



ERLC 



If)'. 
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School of Nursing 
Indiana University 
Indianapolis , Indiana 

Oklahoma State University 
Stillwater, Oklahoma 

Psychology Department 
Miami University 
Oxford, Ohio 

Institute for Research on 

Teaching 
Michigan State University 
Rant Lansing, Michigan 

Stud ent Serv ices 
Richland College 
Dallas, Texas 

Office of Instructional 
development 

University of North Dakota 
Grand Forks, North Dakota 

Bowling Green State University 
Bowling Green, Ohio 

University of California - 

Los Angeles 
Los Angeles, California 

0 ou n s e 1 i ng 0 e n t e r 
University of Missouri-Holla 

Department of Psychology 
University of Michigan 
■\nn Arbor, Michigan 

Col lego of St. Theresa 
W i nona , Mi nneso ta 

r nstitute for Puolio Policy 

Stud i.es 
Vand'M'b i 1 t Un i ves i ty 
Nashv i. Lie, Tennesson 



Instructional Development 
Webster State College 
Ogden, Utah 

Institute for Studies 

in Educational Mathematics 
St. Paul, Minnesota 

College of Arts and Sciences 
University of the Pacific 
Stockton, Cal ifornia 

Institutional Research Office 
U.S. Military Academy 
West Point, New York 

Northwestern University 
Evanston, Illinois 

Office of Educational Research 
St. Olaf College 
Northfieli, Minnesota 

Department of Education 
Oallaudet College 
Washington , DC 

-Arizona Department of Education 
Phoenix , AZ 

University of Iowa 
Iowa City, Iowa 

ornell University Field Study Office 
New York, New York 

Department of Curriculum 
Chicago Public Schools 
Chi cago , Til ino i s 

Pryn Mawr College 

Rryti Mawr , Pennsylvan ia 



ERLC 



Ifi s 
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International 



Director, Applied Research Centre 

St. Clair College 

Windsor, Ontario, Canada N9A634 

Sciences De L' Education 
University of Quebec-Montreal 
Montreal, Quebec, Canada 

Western Australian Institute of Technology 

South Bentley, 6102 

Western Australia, Australia 

Institute of Education 
National University of La not ho 
Loso tho , Africa 

Office for Reseaeh in Academic 
Methods 

Australian National University 
Canberra Act, Australia 

Faculty of Education 
University of British Columbia 
Vancouver, B.C. , Canada 

Educational Development Office 
York University 
Downsview, Ontario , Canada 

Deutsche;* Institut Fur Fernstudien 

An Der Universitat Tubingen 
Tubingen 1, West Germany 

^acuity of Education 
M c 0 i 1 1 University 
Montreal f Quebec , Canad a 

School of Nursing 
University of Ottawa 
Ottawa , Canada 

University of Medina 
"(eg inn Dask, Canada 

Centre for Teaching and Learning 

Services ' 
MeOil 1 Un i vers i ty 
Montreal , Canad'i 
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Alverno College 

3401 S. 39th St. / Milwaukee, Wl 53215 
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